Re: [openstack-dev] [murano] Where can we find an image with Murano Agent
Hi Jerry, Previously these images were available at apps.openstack.org. But since this site is no longer available, the only way to get the images that I know is to build them by yourself. Here is how you can build most basic ubuntu+murano-agent image: https://github.com/openstack/murano-agent/tree/master/contrib/elements Some of the applications in the murano-apps repo require an image with additional elements. They all come with instructions on how to build the image. But, if you are on a recent murano version, you can try to go with a bare ubuntu cloud image since murano should be capable to install agent through the cloud-init without backing it into the image. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis <sla...@mirantis.com> On Fri, Nov 24, 2017 at 11:05 AM, Sun, Yicheng (Jerry) < jerry@windriver.com> wrote: > We are trying to test our Murano deployment. > > We are working with the Pike version of Murano. > > > > Do you know where we can get an image with Murano Agent? > > The community application catalog was taken down. > > We tried following the instructions here: > > https://murano.readthedocs.io/en/latest/image_builders/linux.html > > https://pypi.python.org/pypi/murano-agent/3.3.0 > > > > We were not able to build ubuntu with Murano Agent. > > > > Thanks in advance, > > Jerry > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [murano] MuranoPL types question
Hi Paul,
> Here both key and value appear to be a string (note, I can't confirm this
as the typeinfo function doesn't appear to be available in Instance.yaml
for some reason... perhaps I'm using it wrong)
They are not strings. Reporter converts everything that is passed to it
into string by doing unicode(obj). What you see is the Python string
representation of lists and dicts, where every unicode string is prefixed
with "u".
Typinfo function works on objects (instances of MuranoPL classes), not on
primitive types like strings, dicts and lists.
> 1) Why is the content of this 'get_attr' coming from heat being stored in
the stack outputs as a string,
it is stored exactly as it comes from Heat. In theory conversion to string
could happen on Heat side, but most likely it just the output of reporter
made it look like this
> 2) Is there a way I can cast this to a list of dicts or similar structure
so it can be iterated as expected?
It's hard to answer without seeing your code and how you got the results
that you provided.
> when I access this from a sample app, I end up with a list of strings,
shown by $reporter as: ...
Curly braces in the output indicate that this is not a list of strings but
rather the single dict converted to a string by reporter. So what you see
is the value of that 'vol-7c8ee61f-a444-46a1-ad70-fc6fce6a7b56-attachments'
output, which sounds like what you're wanted it to be. In MuranoPL you can
have very detailed contracts: not just [] (any list) but something like
Contract:
- key1: string().notNull()
key2: int().notNull()
which is a list of dicts with contract on each dict entry. If you don't get
contract violation exception, you can be sure that the list contains list
of dicts with appropriate keys/values rather than list of strings or
anything else
Sincerely yours,
Stan Lagun
Principal Software Engineer @ Mirantis
<sla...@mirantis.com>
On Thu, Mar 30, 2017 at 10:17 AM, Paul Bourke <paul.bou...@oracle.com>
wrote:
> Hi Stan,
>
> I had a quick(hopefully) question about MuranoPL that I hope you might be
> able to help with, Felipe had mentioned you are very knowledgeable in this
> area. If you don't have time please disregard!
>
> I'm working on a patch for the Murano core library to make volume
> attachment info available, which is available at
> https://review.openstack.org/451909. It's mostly working, but I'm having
> some issues getting the types correct in MuranoPL to make this info
> consumable by end users.
>
> The attachment info from a sample run in the stack $outputs looks like
> this (taken from the dashboard using $reporter)
>
> u'vol-7c8ee61f-a444-46a1-ad70-fc6fce6a7b56-attachments':
> u"[{u'server_id': u'2891067b-e7bb-4ab9-a759-9e81096c0682',
> u'attachment_id': u'7456a4b0-3abd-48a0-a0cb-f3fa0f2706bb',
> u'attached_at': u'2017-03-30T16:51:57.00', u'host_name': None,
> u'volume_id': u'373e4a5a-46b6-4091-82d6-b3dba62d552b', u'device':
> u'/dev/vda', u'id': u'373e4a5a-46b6-4091-82d6-b3dba62d552b'}]"
>
> Here both key and value appear to be a string (note, I can't confirm this
> as the typeinfo function doesn't appear to be available in Instance.yaml
> for some reason... perhaps I'm using it wrong)
>
> My goal is to then set this into a property of CinderVolume.yaml, with a
> Contract of '[]'. However, when I access this from a sample app, I end up
> with a list of strings, shown by $reporter as:
>
> (u"[{u'server_id': u'2891067b-e7bb-4ab9-a759-9e81096c0682',
> u'attachment_id': u'f5bd50ab-4040-4e2b-8b50-790781d9bc37',
> u'attached_at': u'2017-03-30T16:51:59.00', u'host_name': None,
> u'volume_id': u'ed7eb535-9e81-4c97-b063-86936d4ee306', u'device':
> u'/dev/vdb', u'id': u'ed7eb535-9e81-4c97-b063-86936d4ee306'}]",)
>
> So my question is:
>
> 1) Why is the content of this 'get_attr' coming from heat being stored in
> the stack outputs as a string, and
>
> 2) Is there a way I can cast this to a list of dicts or similar structure
> so it can be iterated as expected?
>
> Any tips much appreciated.
>
> Thanks,
> -Paul
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] Newton version of Murano fails trying to auto-import a Liberty version of io.murano.applications.zip
It is probably that the package io.murano.applications is not present in the app catalog. io.murano and io.murano.applications are the packages that are bundled with Murano and usually should be preinstalled. You can obtain them here: https://github.com/openstack/murano/tree/stable/newton/meta Regards, Stan On March 7, 2017 at 11:24:48 AM, Waines, Greg (greg.wai...@windriver.com) wrote: I am running a NEWTON version of Murano. When I try to import a package that is using io.murano.applications classes (e.g. SingleServerApplication), it seems to try to auto-import the liberty version of io.murano.applications.zip … and fails. Is this a bug ? Is it suppose to be using NEWTON version of io.murano.applciations.zip ? Actually can’t find NEWTON version, is the URL wrong ? [wrsroot@controller-0 newDemo(keystone_tenant1)]$ murano package-import com.wrs.titanium.murano.examples.afwDemo Error Got non-ok status(404) while connecting to http://apps.openstack.org/api/v1/murano_repo/liberty/apps/io.murano.applications.zip occurred while parsing package io.murano.applications, required by com.wrs.titanium.murano.examples.afwDemo package Inspecting required images Importing package com.wrs.titanium.murano.examples.afwDemo +--+-+--+-++---+-+-+ | ID | Name| FQN | Author | Active | Is Public | Type| Version | +--+-+--+-++---+-+-+ | e7a08d26300e4812b9a5d0f91b2ad430 | APP FW Titanium Murano Demo App | com.wrs.titanium.murano.examples.afwDemo | Greg Waines, Wind River | True | | Application | | +--+-+--+-++---+-+-+ [wrsroot@controller-0 newDemo(keystone_tenant1)]$ let me know, Greg. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] ERROR murano.common.engine [-] yaql.language.exceptions.NoMatchingMethodException: No method "values" for receiver None matches supplied arguments
HOWEVER … for future reference … how do I go about seeing the internal HEAT STACK that Murano builds based on the Murano Application / Packaging ? You can do it be ether 1) Obtaining from the Heat since the stack is accessible and there is an API and UI to get the template 2) Grep murano-engine logs for "Pushing" followed by ay a stack template JSON. Note, that Murano usually does series of stack updates rather than creates it at once. Therefore there going to be several such log records for each deployment Regards, Stan __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] ERROR murano.common.engine [-] yaql.language.exceptions.NoMatchingMethodException: No method "values" for receiver None matches supplied arguments
Greg,
based on the stack trace provided I see that it happened because the Heat
stack output had wrong value.
If you look at
https://github.com/openstack/murano/blob/stable/newton/meta/io.murano/Classes/resources/Instance.yaml#L178-L179
- $netIdToIpsMap: $outputs.get(format('{0}-assigned-ips', $this.name))
- $.ipAddresses: $netIdToIpsMap.values().flatten().distinct() # <---
exception is thrown here
"No method "values" for receiver None..." means that $netIdToIpsMap is null
(None), which, in turn, means that
ether there in no output value with the name "$instanceName-assigned-ips"
or its value is null. In both cases it is not
something that can happen under normal conditions. Unfortunately I cannot
tell why it happened in your case.
I'd start with debugging of the Heat stack in attempt to find out why Heat
didn't set the output values
Regards,
Stan
On March 6, 2017 at 12:02:59 PM, Waines, Greg (greg.wai...@windriver.com)
wrote:
We are attempting to integrate a NEWTON-version of MURANO integrated into
our OpenStack solution.
With a very simple murano package/app, that basically just does a “
$.instance.deploy() “ in its deploy method of its main class,
we are getting the following traceback error (see end of email).
Any thoughts / guidance on potential issues ?
( NOTE … we had been trying to run with the secondary rabbit server, but
now (temporarily) just pointing murano at the core rabbit server to get
past some issues with not having the secondary rabbit server running. )
thanks in advance,
Greg.
2017-03-06 19:28:08.904 184898 ERROR murano.common.engine [-]
yaql.language.exceptions.NoMatchingMethodException: No method "values"
for receiver None matches supplied arguments
Traceback (most recent call last):
File
"/tmp/murano-packages-cache/io.murano/0.0.0/37e6dfe1415c4caa9150a53fc18506ef/Classes/Environment.yaml",
line 120:9 in method deploy of type io.murano.Environment
$.applications.pselect($.deploy())
File
"/tmp/murano-packages-cache/wrs.titanium.murano.examples.cgcs_guest/0.0.0/b7554a5f915748629da6e425005e1933/Classes/cgcs_guest.yaml",
line 41:13 in method deploy of type wrs.titanium.murano.examples.cgcs_guest
$.instance.deploy()
File
"/tmp/murano-packages-cache/io.murano/0.0.0/37e6dfe1415c4caa9150a53fc18506ef/Classes/resources/Instance.yaml",
line 194:9 in method deploy of type io.murano.resources.Instance
$this.endDeploy()
File
"/tmp/murano-packages-cache/io.murano/0.0.0/37e6dfe1415c4caa9150a53fc18506ef/Classes/resources/Instance.yaml",
line 179:24 in method endDeploy of type io.murano.resources.Instance
$netIdToIpsMap.values().flatten().distinct()
File "/usr/lib/python2.7/site-packages/murano/dsl/helpers.py", line 53
in method evaluate
return value(context)
File "/usr/lib/python2.7/site-packages/murano/dsl/yaql_expression.py",
line 85 in method __call__
return self._parsed_expression.evaluate(context=context)
File "/usr/lib/python2.7/site-packages/yaql/language/expressions.py",
line 165 in method evaluate
return self(utils.NO_VALUE, context, self.engine)
File "/usr/lib/python2.7/site-packages/yaql/language/expressions.py",
line 156 in method __call__
return super(Statement, self).__call__(receiver, context, engine)
File "/usr/lib/python2.7/site-packages/yaql/language/expressions.py",
line 37 in method __call__
return context(self.name, engine, receiver, context)(*self.args)
File "/usr/lib/python2.7/site-packages/yaql/language/contexts.py", line
65 in method
data_context, use_convention, function_filter)
File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line
49 in method call
name, all_overloads, engine, receiver, data_context, args, kwargs)
File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line
117 in method choose_overload
args = tuple(arg_evaluator(i, arg) for i, arg in enumerate(args))
File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line
117 in method
args = tuple(arg_evaluator(i, arg) for i, arg in enumerate(args))
File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line
113 in method
and not isinstance(arg, expressions.Constant))
File "/usr/lib/python2.7/site-packages/yaql/language/expressions.py",
line 37 in method __call__
return context(self.name, engine, receiver, context)(*self.args)
File "/usr/lib/python2.7/site-packages/yaql/language/contexts.py", line
65 in method
data_context, use_convention, function_filter)
File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line
49 in method call
name, all_overloads, engine, receiver, data_context, args, kwargs)
File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line
117 in method choose_overload
args = tuple(arg_evaluator(i, arg) for i, arg in enumerate(args))
Re: [openstack-dev] [Murano] Errors on $.instance.deploy() ... WHEN RUNNING WITHOUT MURANO-AGENT RABBIT
py", line 37 in method __call__ return context(self.name, engine, receiver, context)(*self.args) File "/usr/lib/python2.7/site-packages/yaql/language/contexts.py", line 65 in method data_context, use_convention, function_filter) File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line 51 in method call result = delegate() File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line 142 in method return lambda: delegate() File "/usr/lib/python2.7/site-packages/yaql/language/specs.py", line 341 in method func six.iteritems(keyword_args))) File "/usr/lib/python2.7/site-packages/murano/dsl/yaql_functions.py", line 183 in method op_dot return operator(ctx2, receiver, expr) File "/usr/lib/python2.7/site-packages/yaql/language/yaqltypes.py", line 323 in method func new_name, engine, new_receiver, new_context)(*args, **kwargs) File "/usr/lib/python2.7/site-packages/yaql/language/contexts.py", line 65 in method data_context, use_convention, function_filter) File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line 51 in method call result = delegate() File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line 142 in method return lambda: delegate() File "/usr/lib/python2.7/site-packages/yaql/language/specs.py", line 341 in method func six.iteritems(keyword_args))) File "/usr/lib/python2.7/site-packages/murano/dsl/yaql_functions.py", line 183 in method op_dot return operator(ctx2, receiver, expr) File "/usr/lib/python2.7/site-packages/yaql/language/yaqltypes.py", line 323 in method func new_name, engine, new_receiver, new_context)(*args, **kwargs) File "/usr/lib/python2.7/site-packages/yaql/language/contexts.py", line 65 in method data_context, use_convention, function_filter) File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line 51 in method call result = delegate() File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line 142 in method return lambda: delegate() File "/usr/lib/python2.7/site-packages/yaql/language/specs.py", line 341 in method func six.iteritems(keyword_args))) File "/usr/lib/python2.7/site-packages/yaql/standard_library/system.py", line 34 in method op_dot return expr(receiver) File "/usr/lib/python2.7/site-packages/yaql/language/yaqltypes.py", line 269 in method func engine, args, kwargs) File "/usr/lib/python2.7/site-packages/yaql/language/yaqltypes.py", line 236 in method _call result = value(receiver, context, engine) File "/usr/lib/python2.7/site-packages/yaql/language/expressions.py", line 37 in method __call__ return context(self.name, engine, receiver, context)(*self.args) File "/usr/lib/python2.7/site-packages/yaql/language/contexts.py", line 65 in method data_context, use_convention, function_filter) File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line 51 in method call result = delegate() File "/usr/lib/python2.7/site-packages/yaql/language/runner.py", line 142 in method return lambda: delegate() File "/usr/lib/python2.7/site-packages/yaql/language/specs.py", line 341 in method func six.iteritems(keyword_args))) File "/usr/lib/python2.7/site-packages/murano/dsl/yaql_integration.py", line 246 in method payload __receiver, args, kwargs, __context, True), __context) File "/usr/lib/python2.7/site-packages/murano/dsl/murano_method.py", line 230 in method invoke self, this, context, args, kwargs, skip_stub) File "/usr/lib/python2.7/site-packages/murano/dsl/executor.py", line 163 in method invoke_method result = call() File "/usr/lib/python2.7/site-packages/murano/dsl/executor.py", line 153 in method call yaql_engine, context, native_this)(*args, **kwargs) File "/usr/lib/python2.7/site-packages/yaql/language/specs.py", line 60 in method func return self.get_delegate(receiver, engine, context, args, kwargs)() File "/usr/lib/python2.7/site-packages/yaql/language/specs.py", line 341 in method func six.iteritems(keyword_args))) File "/usr/lib/python2.7/site-packages/murano/dsl/yaql_integration.py", line 192 in method payload return body(__self.extension, *args, **kwargs) File "/usr/lib/python2.7/site-packages/murano/engine/system/agent.py", line 71 in method queue_name return self._queue *From: *Stan Lagun <sla...@mirantis.com> *Reply-T
Re: [openstack-dev] [Murano] Errors on $.instance.deploy() ... WHEN RUNNING WITHOUT MURANO-AGENT RABBIT
Greg, you're right in everything you said. This was fixed in master in https://review.openstack.org/#/c/387993/ with intent to fix tests and unblock the gate. But since those tests were introduced in Ocata the fix was not backported to Newton. So I've just did a partial packport of the root cause fix. Here it is: https://review.openstack.org/#/c/441477/ Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis <sla...@mirantis.com> On Fri, Mar 3, 2017 at 11:49 AM, Waines, Greg <greg.wai...@windriver.com> wrote: > Looking for guidance on fixing the following error (end of email) that we > are seeing when ‘deploying’ a very simple murano package/app on a Murano > deployment WITHOUT the second rabbit server for murano-agent. > > > > So we have a NEWTON-version of MURANO integrated into our OpenStack > solution. > > > > We have modified murano.conf: > > … > > # Disallow the use of murano-agent (boolean value) > > disable_murano_agent = true > > … > > > > We are NOT even running the second rabbit server for communication with > murano-agent. > > > > With a very simple murano package/app, that basically just does a “ > $.instance.deploy() “ in its deploy method of its main class. > > *( NOTE: I have tested this murano package/app on other Murano setups and > it works … although those Murano setups had murano-agent and second rabbit > enabled. )* > > > > And we get the following traceback error (see end of email). > > > > Questions: > > - I believe this is a valid configuration > > - but does anyone actually run Murano in this type of setup ? I am > guessing not many … so is this an upstream murano bug that I am seeing ? … > that nobody else sees because basically no one runs in this mode ? > > > > - if this should work, and works in other setups, > > … any guidance on how to resolve / debug what is going on here ? > > > > > > Thanks in advance, > > Greg. > > > > > > > > 2017-03-03 17:46:35.175 12568 ERROR murano.common.engine [-] > > AttributeError: 'AgentListener' object has no attribute '_results_queue' > > Traceback (most recent call last): > > File "/tmp/murano-packages-cache/io.murano/0.0.0/ > 3317e706ecd1417bb748361a6a3385d2/Classes/Environment.yaml", line 120:9 in > method deploy of type io.murano.Environment > > $.applications.pselect($.deploy()) > > File "/tmp/murano-packages-cache/wrs.titanium.murano.examples. > VmFip_NoAppDeploy/0.0.0/829a861c408a4516b0589d04cce232 > 48/Classes/VmFip_NoAppDeploy.yaml", line 41:13 in method deploy of type > wrs.titanium.murano.examples.VmFip_NoAppDeploy > > $.instance.deploy() > > File "/tmp/murano-packages-cache/io.murano/0.0.0/ > 3317e706ecd1417bb748361a6a3385d2/Classes/resources/Instance.yaml", line > 193:9 in method deploy of type io.murano.resources.Instance > > $this.beginDeploy() > > File "/tmp/murano-packages-cache/io.murano/0.0.0/ > 3317e706ecd1417bb748361a6a3385d2/Classes/resources/Instance.yaml", line > 131:28 in method beginDeploy of type io.murano.resources.Instance > > $.prepareUserData() > > File "/tmp/murano-packages-cache/io.murano/0.0.0/ > 3317e706ecd1417bb748361a6a3385d2/Classes/resources/LinuxMuranoInstance.yaml", > line 14:19 in method prepareUserData of type io.murano.resources. > LinuxMuranoInstance > > $.generateUserData() > > File "/tmp/murano-packages-cache/io.murano/0.0.0/ > 3317e706ecd1417bb748361a6a3385d2/Classes/resources/LinuxMuranoInstance.yaml", > line 81:31 in method generateUserData of type io.murano.resources. > LinuxMuranoInstance > > $region.agentListener.queueName() > > File "/usr/lib/python2.7/site-packages/murano/dsl/helpers.py", line > 58 in method evaluate > > for d_key, d_value in six.iteritems(value)) > > File "/usr/lib/python2.7/site-packages/yaql/language/utils.py", line > 122 in method __init__ > > self._d = dict(*args, **kwargs) > > File "/usr/lib/python2.7/site-packages/murano/dsl/helpers.py", line > 58 in method > > for d_key, d_value in six.iteritems(value)) > > File "/usr/lib/python2.7/site-packages/murano/dsl/helpers.py", line > 53 in method evaluate > > return value(context) > > File "/usr/lib/python2.7/site-packages/murano/dsl/yaql_expression.py", > line 85 in method __call__ > > return self._parsed_expression.evaluate(context=context) > > File "/usr/lib/python2.7/site-packag
Re: [openstack-dev] [heat][yaql] Deep merge map of lists?
There is a bug in how yaql interprets its 3rd argument (aggregator). It was supposed to be a function to aggregate values within each group but instead was applied to the outcome of groupBy. I submitted a fix for this: https://review.openstack.org/363191. Though I'm not sure we can release new yaql version that late in the Newton cycle. Meanwhile it is better to use one of alternative solutions above so that it won't break after this patch get merged. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis <sla...@mirantis.com> On Tue, Aug 30, 2016 at 12:46 PM, Thomas Herve <the...@redhat.com> wrote: > On Tue, Aug 30, 2016 at 6:02 PM, Steven Hardy <sha...@redhat.com> wrote: > > On Tue, Aug 30, 2016 at 04:10:47PM +0200, Jiří Stránský wrote: > >> > >> On 30.8.2016 10:17, Steven Hardy wrote: > >> > >> > >> > >> > Yeah, that gets us closer, but we do need to handle more than one > value > >> > (list entry) per key, e.g: > >> > > >> > data: > >> >l: > >> > - "gnocchi_metricd_node_names": ["a0", "a1", "a2"] > >> >"tripleo_packages_node_names": ["a0", "a1", "a2"] > >> > - "nova_compute_node_names": ["b0"] > >> >"tripleo_packages_node_names": ["b0"] > >> > > >> > Output needs to be like: > >> > > >> > "gnocchi_metricd_node_names": ["a0", "a1", "a2"] > >> > "tripleo_packages_node_names": ["a0", "a1", "a2", "b0"] > >> > "nova_compute_node_names": ["b0"] > >> > > >> > >> Hoping this could do it: > >> > >> [stack@instack ~]$ cat yaq.yaml > >> heat_template_version: 2016-10-14 > >> > >> outputs: > >> debug: > >> value: > >> yaql: > >> expression: $.data.l.reduce($1.mergeWith($2)) > >> data: > >> l: > >> - "gnocchi_metricd_node_names": ["a0", "a1", "a2"] > >> "tripleo_packages_node_names": ["a0", "a1", "a2"] > >> - "nova_compute_node_names": ["b0"] > >> "tripleo_packages_node_names": ["b0"] > > > > Thanks for this! > > > > Unfortunately I dont think it works with more than two list items: > > > > debug_tripleo2: > > value: > > yaql: > > expression: $.data.l.reduce($1.mergeWith($2)) > > data: > > l: > > - "gnocchi_metricd_node_names": ["overcloud-controller-0", > > "overcloud-controller-1", "overcloud-controller-2"] > > "tripleo_packages_node_names": ["overcloud-controller-0", > "overcloud-controller-1", "overcloud-controller-2"] > > - "nova_compute_node_names": ["overcloud-compute-0"] > > "tripleo_packages_node_names": ["overcloud-compute-0"] > > "tripleo_packages_node_names2": ["overcloud-compute-0"] > > - "ceph_osd_node_names": ["overcloud-cephstorage-0"] > > "tripleo_packages_node_names": ["overcloud-cephstorage-0"] > > "tripleo_packages_node_names2": > ["overcloud-cephstorage-0"] > > > > $ heat output-show foo5 debug_tripleo2 > > stack output show" instead > > Output error: can only concatenate tuple (not "list") to tuple > > > > I've not dug too deeply yet, but assuming that's a yaql error vs a heat > bug > > it looks like it won't work. > > I'd say it's a yaql bug (commented on the bug opened in Heat), that > can work around using the list_merger argument: > > $.data.l.reduce($1.mergeWith($2, listMerger => $1.toList() + $2.toList())) > > Still slightly more elegant than the one I came up with. > > -- > Thomas > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] Developer's productivity with MuranoPL
Hi Alex,
I'm two hands up for making developers life easier. However we should
remember that MuranoPL is not a scripting language and not a Python
replacement. There's very little point in writing Hello Word on MuranoPL.
And once you go for even the simplest app possible you will still have to
have OpenStack installed, manifest file written, class created etc. So the
only use case that I see is to type some simple statement that doesn't
involve any deployment or other I/O
and see how it works. In 80% of cases the same could be done in yaql CLI
tool. That's why I don't expect it to be that much of a productivity boost.
IMO debugger will do much better. And we could have an expression
evaluation functionality within debugger.
Btw, most of the steps you mentioned can be solved by an application stub
generator and configuration setting that will make murano take packages
from the local folder
Sincerely yours,
Stan Lagun
Principal Software Engineer @ Mirantis
<sla...@mirantis.com>
On Tue, Aug 16, 2016 at 7:31 AM, Alexander Tivelkov <ativel...@mirantis.com>
wrote:
> Hi folks,
>
> The developer's productivity and ease of use is critical for the language
> adoption, and I believe we have issues with these in MuranoPL. The things
> are slowly getting better (thanks to the guys who are bulding th MuranoPL
> syntax checker [1] which will be a very useful tool), but that's not
> enough: we need more tools to simplify dev's life.
>
> Let me provide a quick example: what does it take to run a "Hello World"
> in MuranoPL?
> To create such an introductory MuranoPL program the developer has to:
> 1) Install Murano with the rest of openstack (i.e. spin up devstack with
> Murano and Glare plugins)
> 2) Write the actual HelloWorld "program" (mind the absence of simple
> "print" function and the non-so-obvious need to extend
> io.murano.Application class for the example to be runnable as murano's
> entry point)
> 3) Write the UI.yaml file to provide the input object graph (even if there
> is no user-supplied paramters - we still have to have this file)
> 4) Write a manifest.yaml
> 5) zip the program, the ui and the manifest into a package
> 6) Upload the package to Glare
> 7) Create an environment in Murano dashboard
> 8) Add the demo app to the environment
> 9) Deploy the environment
>
> I hope I din't miss any step. Even excluding the time to spin up devstack
> (and the high probability that a newcomer will not do that properly from
> the first attempt) this is going to take at least 30 minutes. Even when the
> environment is set up, the whole "make code changes - recreate the package
> - reupload the package - recreate the environment - redepoy" cycle is
> cumbersome.
>
> What do we need is a simple and easy way to run MuranoPL code without the
> need to set up all the server-sides components, generate object model and
> interact with all the production-grade machinery murano has under the hood.
> To do something like:
>
>$ cat "- print('Hello, World')" > ./hello.yaml
>$ murano-pl ./hello.yaml
>
> Ideally there should be an interactive REPL-like shell, with smart
> indentation and code completion similar to the Python's (we have one for
> yaql, and so we should for MuranoPL).
>
> That is not a very hard thing to do, and it will simplify the developers'
> onboarding dramatically.
>
> So, I propose to start with a simple thing: to separate the MuranoPL
> interpreter (mostly the code located in murano.dsl package, plus some other
> stuff) from the rest of Murano. Put it into a standalone reporitory, so it
> may be packaged and distributed independently. The developers will just
> need to 'pip install murano-pl' to have the local interpreter without all
> the dependencies on murano api, engine, database etc. This new package may
> include the murano-pl test-runner (this tool is currently part of the main
> murano repo and is hard to use since it requires to have a valid config
> file which is not a proper option for a cli tool). Then we may include some
> other devloper-side tools, such as a murano-pl debugger (when we finally
> have one: this is a separate topic).
> Finally, we will need to remove the core library (murano.engine.system)
> package from the main murano repo and also make it a standalone
> repo/package with its own lifecycle (it would be very helpfull if we could
> release/update the core library more frequently). So the main murano repo
> (and appropriate package) will contain only the server-side murano
> components: the REST API, the engine, DB api, model and migrations etc.
>
> When this is done we may begin adding other developer's productivity
> tools: starting with debugger and then to various kinds of IDE
> integrations.
Re: [openstack-dev] NoMethodRegisteredException when deploying Murano environment
Hi, most likely it is caused by using wrong version of the core library package. Some parts of it are written in Python rather than in MuranoPL and located inside murano source codes. So if you use newer version of core library with older version of Murano you get this error. In particular it cannot find https://github.com/openstack/murano/blob/stable/mitaka/murano/engine/system/net_explorer.py#L195 method which is available since mitaka. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis <sla...@mirantis.com> On Wed, Aug 24, 2016 at 4:37 AM, Kirill Zaitsev <kzait...@mirantis.com> wrote: > Hi, it’s impossible to say without knowing what application you’re trying > to deploy. Also this mailing list is not the best place for usage questions > =) Please come to our IRC channel #murano — we would try to help you find > what went wrong. > > -- > Kirill Zaitsev > Murano Project Tech Lead > Software Engineer at > Mirantis, Inc > > On 24 août 2016 at 05:57:32, wuhan (wuhan9...@163.com) wrote: > > listNeutronExtensions > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [mistral] [murano] [yaql] yaqluator bug
Hi!
The issue with join is just a yaql bug that is already fixed. The problem
with yaqluator is that it doesn't use the latest yaql library.
Another problem is that it does't sets options correctly. As a result it is
possible to bring the site down with a query that produces endless
collection
Sincerely yours,
Stan Lagun
Principal Software Engineer @ Mirantis
<sla...@mirantis.com>
On Tue, Jun 28, 2016 at 9:46 AM, Elisha, Moshe (Nokia - IL) <
moshe.eli...@nokia.com> wrote:
> Hi,
>
> Thank you for the kind words, Alexey.
>
> I was able to reproduce your bug and I have also found the issue.
>
> The problem is that we did not create the parser with the engine_options
> used in the yaql library by default when using the CLI.
> Specifically, the "yaql.limitIterators" was missing… I am not sure that
> this settings should have this affect but maybe the Yaql guys can comment
> on that.
>
> If we will change yaqluator to use this setting it will mean that
> yaqluator will not be consistent with Mistral because Mistral is using YAQL
> without this engine option (If I use your example in a workflow, Mistral
> returns exactly like the yaqluator returns)
>
>
> Workflow:
>
> ---
> version: '2.0'
>
> test_yaql:
> tasks:
> test_yaql:
> action: std.noop
> publish:
> output_expr: <% [1,2].join([3], true, [$1, $2]) %>
>
>
> Workflow result:
>
>
> [root@s53-19 ~(keystone_admin)]# mistral task-get-published
> 01d2bce3-20d0-47b2-84f2-7bd1cb2bf9f7
> {
> "output_expr": [
> [
> 1,
> 3
> ]
> ]
> }
>
>
> As Matthews pointed out, the yaqluator is indeed OpenSource and
> contributions are welcomed.
>
> [1]
> https://github.com/ALU-CloudBand/yaqluator/commit/e523dacdde716d200b5ed1015543d4c4680c98c2
>
>
>
> From: Dougal Matthews <dou...@redhat.com>
> Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Date: Monday, 27 June 2016 at 16:44
> To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> Subject: Re: [openstack-dev] [mistral] [murano] [yaql] yaqluator bug
>
> On 27 June 2016 at 14:30, Alexey Khivin <akhi...@gmail.com> wrote:
>
>> Hello, Moshe
>>
>> Tomorrow I discovered yaqluator.com for myself! Thanks for the useful
>> tool!
>>
>> But suddenly I was said that the expression
>> [1,2].join([3], true, [$1, $2])
>> evaluated to [[1,3]] on the yaqluator
>>
>> A the same time this expression evaluated right when I using raw yaql
>> interpreter.
>>
>> Could we fix this issue?
>>
>> By the way, don't you want to make yaqluator opensource? If you would
>> transfer yaqluator to Openstack Foundation, then community will be able to
>> fix such kind of bugs
>>
>
> It looks like it is open source, there is a link in the footer:
> https://github.com/ALU-CloudBand/yaqluator
>
>
>>
>> Thank you!
>> Best regards, Alexey Khivin
>>
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] Does Murano support version management?
Hi, Murano does have partial support for version management. Murano (deployment) engine can work with several versions of the same package. You can even have them in single environment. I order to store several such packages you must use Glare backend rather than old API backend and must use recent Murano version. But there is no support in UI yet. So if you plan to use CLI or API for automation then it is doable. However if you want to choose package version in the dashboard then we are not there yet. Also note that the package version is for the Murano application, not for the software that this app installs. So if you want to let user choose what version of the software to install this can be either one of application's inputs or there can be several independent Murano apps (one per version + library package to put common code there) See [1] and [2] for more information. [1]: http://docs.openstack.org/developer/murano/draft/appdev-guide/murano_pl.html#versioning [2]: http://murano-specs.readthedocs.io/en/latest/specs/liberty/murano-versioning.html Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis <sla...@mirantis.com> On Tue, May 31, 2016 at 10:20 PM, Jay Lau <jay.lau@gmail.com> wrote: > Hi, > > I have a question for Murano: Suppose I want to manage two different > version Spark packages, does Murano can enable me create one Application in > application catalog but can enable me select different version spark > packages to install? > > -- > Thanks, > > Jay Lau (Guangya Liu) > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [murano] Nominate Victor Ryzhenkin to Murano Core
+1! Well deserved! Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis On Tue, Jan 26, 2016 at 5:20 AM, Yang, Lin A <lin.a.y...@intel.com> wrote: > Glad to see this happened. :) > Well deserved, Victor. > > Lin Yang > @Intel > >> On Jan 23, 2016, at 01:33, Serg Melikyan <smelik...@mirantis.com> wrote: >> >> I would like to nominate Victor Ryzhenkin (freerunner on IRC) join to the >> Murano >> core-reviewers team. He has been an active member of the Murano team >> for some time now. You can check his review activity report here: >> http://stackalytics.com/report/contribution/murano-group/90 >> >> Team please vote for this change in reply to this message. >> -- >> Serg Melikyan, Senior Software Engineer at Mirantis, Inc. >> http://mirantis.com | smelik...@mirantis.com >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Murano] Murano actions authentication and accessability
Hello everyone! As many of you know, Murano actions is the Murano way to do something to applications that are already deployed using simple ReST API. For example, health monitoring system may trigger an action on high CPU load so that application could scale itself up. However there are several obstacles that makes it hard to use actions for such scenarios: * There is no way to obtain a URI for the action from within MuranoPL code. If application wants that health monitoring system to call Murano back it need to provide it with a URI it could use to signal the event (i.e. URI of the action). However applications are not aware of Murano API server endpoint and other required details which makes it hard to construct such a URI from MuranoPL. This effectively puts a constraint that the action caller must be murano-aware which is a rare case yet. * Even if application manages somehow to provide external system with an action URI, an attempt to call it will require a valid Keystone token. It is because action URIs are served by Murano API which requires it and especially because all changes made to applications must be authorized. This may be a problem, especially when that external system is not OpenStack-aware and cannot authenticate to Keystone. Recently I submitted a specification for review that addresses those issues. It can be fond here: [1]. It is not perfect and it adds some pieces of code to Murano that I wish could be handled by Keystone alone. However it should solve most of use cases we have today. I'd like to ask anyone who faced those issues or have an experience in the field to read and share your feedback on the spec. We need to make sure that it doesn't have serious flaws and does cover your use cases before we proceed with implementation. Thanks for your help! [1]: https://review.openstack.org/#/c/251870/ Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [mistral] [murano] [yaql] An online YAQL evaluator [was RE: [mistral] [murano] An online YAQL evaluator]
Very cool, indeed! Even myself (developer of yaql) often use yaqluator instead of yaql CLI :) Finally took a time to see source codes. Few tips that you may find useful: * The most expensive operation in yaql is YaqlFactory.create(). This method builds a parser and it takes significant time (orders of magnitude more then expression parsing). I see that you create factory/parser on each evaluation. However parser was designed to be immutable and thread safe so it may be reasonable to create it only once (+1 for legacy mode) and reuse it across queries. This will make a big difference if the site become popular * You don't use engine options. There are number of useful settings that allow to limit iterator length or memory usage. For example looks like I killed the site by executing [1, 2].cycle() which returns indefinite collection. Sorry for that. Another useful option is the one that converts sets to lists on output. Without it set function cannot be used in yaqluator. Also see [1] [1]: https://review.openstack.org/#/c/268673/ Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis On Wed, Jan 13, 2016 at 9:17 AM, Renat Akhmerov <rakhme...@mirantis.com> wrote: > Very cool! It now even supports code assistance :) > > Moshe, can’t stop saying good words :) Amazing! > > Renat Akhmerov > @ Mirantis Inc. > > > > On 22 Dec 2015, at 23:06, ELISHA, Moshe (Moshe) > <moshe.eli...@alcatel-lucent.com> wrote: > > Hi all, > > Thank you for the kind words. > Just wanted to let you know that yaqluator[1] was updated and now supports > YAQL 1.0.2. > There is also a checkbox there to work in “Legacy Mode”. > > Hope you will find it useful. > > [1] http://yaqluator.com/ > > > From: Renat Akhmerov [mailto:rakhme...@mirantis.com] > Sent: Friday, August 14, 2015 11:51 AM > To: OpenStack Development Mailing List (not for usage questions) > Subject: Re: [openstack-dev] [mistral] [murano] An online YAQL evaluator > > I just read this thread so decided to add my 2 cents into the collection of > opinions. > > Guys, I tried it out a couple of weeks ago (was told about it by one of my > colleagues). This is really incredible! Especially given that you completed > it in 24 hours :) I think as YAQL attracts more and more users it will be > very handy tool. I am actually for improving it further. > > Thanks a lot! Looking forward to switch to yaql 1.0! > > Renat Akhmerov > @ Mirantis Inc. > > > > > On 05 Aug 2015, at 04:09, Stan Lagun <sla...@mirantis.com> wrote: > > Dmitry, > > yaql 1.0 has both str() and len() and much much more so there is no need to > support them explicitly since Mistral is going to switch to yaql 1.0 and > yaqluator.com is going to do the same > > Sincerely yours, > Stan Lagun > Principal Software Engineer @ Mirantis > > > > On Tue, Aug 4, 2015 at 8:55 PM, Dmitri Zimine <dzim...@stackstorm.com> > wrote: > Awesome! Really. > > Thank you folks for doing this. > > I am so much looking forward to moving it to 1.0 with more built-in > functions and more power to extend it... > > Note that Mistral has a few extensions, like `str`, `len`, which are not in > the scope of evaluator. > > DZ> > > > On Aug 2, 2015, at 12:44 PM, Stan Lagun <sla...@mirantis.com> wrote: > > > Guys, this is awesome!!! > > Happy to see yaql gets attention. One more initiative that you may find > interesting is https://review.openstack.org/#/c/159905/ > This is an attempt to port yaql 1.0 from Python to JS so that the same can > be done in browser > > Sincerely yours, > Stan Lagun > Principal Software Engineer @ Mirantis > > > > On Sun, Aug 2, 2015 at 5:30 PM, Nikolay Makhotkin <nmakhot...@mirantis.com> > wrote: > Hi guys! > > That's awesome! It is very useful for all us! > > -- > Best Regards, > Nikolay > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.
Re: [openstack-dev] [Murano] How to debug a failed deployment?
Vahid, The bug you mentioned was fixed recently: https://review.openstack.org/#/c/255154/ Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis <sla...@mirantis.com> On Fri, Dec 11, 2015 at 10:46 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Nikolay, > > Is there further information I could provide about the bug? > > Thanks. > --Vahid > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [murano] Is there blueprint to add argument collection UI for action
Tony, I forgot to mention that it possible to call actions with parameters using API. It is just a UI that is missing. Also I'd like to mention that any involvement of yours will be appreciated - ideas, suggestions, commits or even if you take a lead over this feature - everything will help to make it be available sooner than later. Please let me know if I can help. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis <sla...@mirantis.com> On Mon, Dec 7, 2015 at 10:25 AM, WANG, Ming Hao (Tony T) < tony.a.w...@alcatel-lucent.com> wrote: > Stan, > > > > Thanks for your information very much. > > I’m looking forward for this feature! J > > > > Thanks, > > Tony > > > > *From:* Stan Lagun [mailto:sla...@mirantis.com] > *Sent:* Saturday, December 05, 2015 4:52 AM > *To:* WANG, Ming Hao (Tony T) > *Cc:* OpenStack Development Mailing List (not for usage questions) > *Subject:* Re: [murano] Is there blueprint to add argument collection UI > for action > > > > Tony, > > > > Here is the blueprint: > https://blueprints.launchpad.net/murano/+spec/action-ui > > > > The problem is much bigger then it seems to. Action parameters are very > similar to regular application properties and thus there should be a > dedicated dynamic UI form (think ui.yaml) for each action describing its > inputs. > > We discussed it many times and each time the resolution was that it is > better to do major refactoring of dynamic UI forms before introducing > additional forms. The intention was to either simplify or completely get > rid of ui.yaml because it is yet another DSL to learn. Most of the > information from ui.yaml could be obtained by examining MuranoPL class > properties. And what is missing could be added as additional metadata right > to the classes. However a lot of work requires to do it properly (starting > from new API that would be MuranoPL-aware). That's why we still have no > proper UI for the actions. > > > > Maybe we should reconsider and to have many ui forms in single package > until we have a better solution. > > > > > > > Sincerely yours, > Stan Lagun > Principal Software Engineer @ Mirantis > > > > On Fri, Dec 4, 2015 at 5:42 AM, WANG, Ming Hao (Tony T) < > tony.a.w...@alcatel-lucent.com> wrote: > > Dear Stan and Murano developers, > > > > The current murano-dashboards can add the action button for the actions > defined in Murano class definition automatically, which is great. > > Is there any blueprint to add argument collection UI for the action? > > Currently, the murano-dashboards only uses environment_id + action_id to > run the actions, and user has no way to provide action arguments from > UI. > > > > Thanks in advance, > > Tony > > > > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] How to debug a failed deployment?
Vahid, The main log file that is missing is murano-engine log. It is murano-engine who does the deployment. There are also no agent logs because agents are on VMs and none of them were created. >From what I see from you screenshots deployment task doesn't even reaches murano-engine. There can be 2 possible reasons for this: 1) murano-engine is not running 2) incorrect configuration of oslo.messaging. Murano API sends tasks to murano-engine through RabbitMQ using oslo.messaging library. Settings in your config file differ from those in mine. For example in my config file rabbit_host is in [DEFAULT] section while in yours it is in [oslo_messaging_rabbit]. Maybe it is okay and my config is outdated. Maybe you use newer version of oslo.messaging that has different config or it can be configured in several ways. I'm not sure. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis <sla...@mirantis.com> On Fri, Dec 4, 2015 at 8:54 AM, Vahid S Hashemian <vahidhashem...@us.ibm.com > wrote: > Dmytro, Kate, Stan, > > Thank you for your responses. I'll try to provide more information about > my environment and the deployment in this note. > Hopefully, this will give you some clue about what the issue is with my > setup. > >1. The murano config file I am using is at >http://paste.openstack.org/show/480855/ >2. The HOT package I am trying to deploy is attached >3. A sample of log entries I see in murano log is at >http://paste.openstack.org/show/480854/ >4. No heat stack is created at all. As I mentioned in my previous >message, it's as if murano cannot talk to heat. >5. Heat engine logs are at http://paste.openstack.org/show/480856/ >6. Heat API logs are at http://paste.openstack.org/show/480857/ >7. Some screen shots are also attached that show the status of >deployment more than 30 minutes after start. Existing heat stacks show that >the template by itself was deployed to heat without an issue. >8. I couldn't find murano agent logs. Please let me know where I can >find them and I'll provide those too. > > > > Regards, > --Vahid > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [murano] Is there blueprint to add argument collection UI for action
Tony, Here is the blueprint: https://blueprints.launchpad.net/murano/+spec/action-ui The problem is much bigger then it seems to. Action parameters are very similar to regular application properties and thus there should be a dedicated dynamic UI form (think ui.yaml) for each action describing its inputs. We discussed it many times and each time the resolution was that it is better to do major refactoring of dynamic UI forms before introducing additional forms. The intention was to either simplify or completely get rid of ui.yaml because it is yet another DSL to learn. Most of the information from ui.yaml could be obtained by examining MuranoPL class properties. And what is missing could be added as additional metadata right to the classes. However a lot of work requires to do it properly (starting from new API that would be MuranoPL-aware). That's why we still have no proper UI for the actions. Maybe we should reconsider and to have many ui forms in single package until we have a better solution. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis <sla...@mirantis.com> On Fri, Dec 4, 2015 at 5:42 AM, WANG, Ming Hao (Tony T) < tony.a.w...@alcatel-lucent.com> wrote: > Dear Stan and Murano developers, > > > > The current murano-dashboards can add the action button for the actions > defined in Murano class definition automatically, which is great. > > Is there any blueprint to add argument collection UI for the action? > > Currently, the murano-dashboards only uses environment_id + action_id to > run the actions, and user has no way to provide action arguments from > UI. > > > > Thanks in advance, > > Tony > > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] How to debug a failed deployment?
Vahid, The only reason HOT deployment could last forever is that Heat stack never enters Ready of Failed state. Please check what is the Heat stack status. I guess it is not Murano issue (except for the fact that we don't provide timeout for HOT deployment :) ) Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis <sla...@mirantis.com> On Thu, Dec 3, 2015 at 11:54 AM, Ekaterina Chernova <efedor...@mirantis.com> wrote: > Hi! > > Murano agent doesn't involved in the deployment of hot packages, that's > why this setting doesn't help. > > Please attach agent logs and heat logs also (check if heat stack is > created or not). > > Regards, Kate. > > > Regards, > Kate. > > > On Thu, Dec 3, 2015 at 11:44 AM, Dmytro Dovbii <ddov...@mirantis.com> > wrote: > >> Hi Vahid, >> >> Could you provide your logs anyway? Please publish them to >> http://paste.openstack.org/ for example >> >> 2015-12-02 23:08 GMT+02:00 Vahid S Hashemian <vahidhashem...@us.ibm.com>: >> >>> Hello, >>> >>> I am trying to deploy a hello world HOT package in murano but the >>> deployment runs forever and does not stop (does not succeed or fail). >>> Stan advised me to modify the config setting below to set a timeout: >>> >>> # Time for waiting for a response from murano agent during the >>> # deployment (integer value) >>> agent_timeout = 300 >>> >>> As you see I set the timeout to 300 seconds and restarted murano API, >>> but even after 15 minutes my deployment is still going. >>> I think my environment is not configured correctly for this to happen. >>> >>> I have devstack running on a VM, and my murano development is done on a >>> separate VM. >>> I bring up murano API and UI on my dev VM that successfully connect to >>> my devstack VM. >>> >>> I am able to deploy the HOT yaml using heat without an issue. >>> From what I can see it seems that murano is not able to talk to heat, >>> because I don't see any stack being created as a result of my deployment. >>> >>> I have also enabled murano logging in the config file as below: >>> >>> # (Optional) Name of log file to output to. If no default is set, >>> # logging will go to stdout. (string value) >>> # Deprecated group/name - [DEFAULT]/logfile >>> log_file = /opt/stack/logs/murano.log >>> >>> But no error is reported while the deployment is in progress. >>> >>> Any tips on how to force the timeout, and also on how to resolve the >>> deployment problem is appreciated. >>> >>> Regards, >>> --Vahid >>> >>> >>> __ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [murano]How to use Murano to transmit files to Mistral and execute scripts on Mistral
On Mon, Nov 30, 2015 at 5:07 AM, WANG, Ming Hao (Tony T) < tony.a.w...@alcatel-lucent.com> wrote: > For the object storage support, does Murano have any plan to support the > auto-uploading function? Murano has plans to support everything applications may need. It is just a matter of priorities/schedule/time etc. It will happen much sooner with your help/contributions. Currently AFAIK there is no blueprint for this feature so nobody developing it Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis <sla...@mirantis.com> __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [murano]How to use Murano to transmit files to Mistral and execute scripts on Mistral
Hi Tony, Technically your solution is possible. Murano agent can be installed manually. All queue names are in agent's config file. Murano can talk to that manually installed agent and put files to that server. However I strongly advice not to do that. First of all it is not secure. Murano agent was designed as a remote execution mechanism rather than a tool to transfer files. Also it was design to be installed on VM rather that on a shared server. Currently there is no authentication for agents. Although it is not truly secure it still provides significant level of security because to make use of the agent you need to know its queue name which is a long random string that only murano engine and agent know and AMQP doesn't provide an API to query list of queue names. But if you install agent manually there will be a single queue. And all applications that need to access that agent will also need to know queue name. Thus it can be easily be compromised and attackers will gain ability to execute arbitrary code on the nova server. It is also doesn't seem like a right solution. There should not be such specific requirements like this to perform such trivial thing. Especially if it involves using tool that was design for something else. So what is the right solution? I can see 2 possible ways to address this issue: 1. Improve Mistral API to support file attachments to its workbooks (if it doesn't support it yet). Improve murano-mistral integration to support this feature. 2. Implement integration of Murano with object storage. Thus it will be possible from Murano workflow to upload file and provide Mistral with a URI to it. Solution #2 seem to be better. At least because it opens new possibilities for all applications and not just those that use Mistral. There is also a workaround that I can think of: make Murano application create temporary VM with FTP server or something like that, put file into it and give Mistral a URI it could use to access the file. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis <sla...@mirantis.com> On Wed, Nov 18, 2015 at 1:06 PM, WANG, Ming Hao (Tony T) < tony.a.w...@alcatel-lucent.com> wrote: > Dear Alexander and Murano developers and testers, > > > > Any suggestion for this? J > > > > Thanks, > > Tony > > > > *From:* WANG, Ming Hao (Tony T) > *Sent:* Tuesday, November 17, 2015 6:12 PM > *To:* OpenStack Development Mailing List (not for usage questions) > *Subject:* RE: [openstack-dev] [murano]How to use Murano to transmit > files to Mistral and execute scripts on Mistral > > > > Alexander, > > > > Thanks for your response. > > During the workflow running stage, it may need to access some other > artifacts. > > In my case, I use Mistral workflow to call Ansible playbook, and I need > Murano to put Ansible playbook into right place on Mistral server so that > Mistral workflow can find it. > > > > Thanks, > > Tony > > > > *From:* Alexander Tivelkov [mailto:ativel...@mirantis.com > <ativel...@mirantis.com>] > *Sent:* Tuesday, November 17, 2015 4:19 PM > *To:* OpenStack Development Mailing List (not for usage questions) > *Subject:* Re: [openstack-dev] [murano]How to use Murano to transmit > files to Mistral and execute scripts on Mistral > > > > Hi Tony, > > > > Probably I am missing something, but why do you need Murano Agent to > interact with Mistral? You can call Mistral APIs right from MuranoPL code > being executed by Murano Engine. Murano's core library contains the > io.murano.system.MistralClient > class ([1]) which may be used to upload and run mistral workflows. > > > > Please let me know if you need more details on this > > > > [1] > https://github.com/openstack/murano/blob/master/murano/engine/system/mistralclient.py > > > > On Tue, Nov 17, 2015 at 5:07 AM WANG, Ming Hao (Tony T) < > tony.a.w...@alcatel-lucent.com> wrote: > > Dear Murano developers and testers, > > > > I want to put some files that Mistral workflow needs into Murano package, > and hope Murano can transmit them to Mistral before it calls Mistral > workflow. > > The flow should be as following: > > 1. User uploads one Murano package which includes both > Murano artifacts and Mistral artifacts to Murano; > > 2. Murano transmits the Mistral artifacts to Mistral, > and Mistral does its work. > > > > After study, I thought muranoagent may be helpful and plan to install a > muranoagent on the Mistral server since it can put files into nova server, > and can run scripts on the nova server. > > After further study, I found muranoagent solution may be not feasible: > > 1. muranoagent and m
Re: [openstack-dev] [Murano] 'NoMatchingFunctionException: No function "#operator_." matches supplied arguments' error when adding an application to an environment
Here is the full story: real YAML that is generated is stored in murano database in packages table. Murano Dashboard obtains form definitions in YAML from API. But to improve performance it also caches them locally. And when it does it stores them in Python pickle format [1] rather then original YAML, but doesn't change the extension (actually this is a bug). That's why when you take yamls from dashboard cache they doesn't look like a valid YAML though they contain the same information and can be decrypted back (that's what I did to find the error you had). But is is much easier to take real YAML from database though [1]: https://docs.python.org/2/library/pickle.html Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis <sla...@mirantis.com> On Fri, Nov 27, 2015 at 1:25 AM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Thanks Stan for the pointer. > > I removed the line that referred to the 'name' property and now my > application is added to the environment without any errors. > However, what I see in ui.yaml still doesn't look like YAML. > > I'm attaching samples again. > > > Even for HOT packages the content is not YAML. > > Regards, > --Vahid > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] 'NoMatchingFunctionException: No function "#operator_." matches supplied arguments' error when adding an application to an environment
Vahid, I see what is the problem. You are generating UI form that looks like this: Forms: - group0: fields: [] Application: name: $.group0.name ?: type: "io.murano.apps.generated.CsarHelloWorld" Version: 2.2 The problem in in "name: $.group0.name". Previously you removed the "name" field from the form because it is provided automatically by the 2.2 form format. However you did not removed the expression that was referring to it. So just drop this expression from the form (and form generation code). If you will need name during deployment time you always can obtain it using name($this) YAQL expression as here: https://github.com/openstack/murano/blob/master/contrib/plugins/cloudify_plugin/cloudify_applications_library/Classes/CloudifyApplication.yaml#L46 Hope this will help Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis <sla...@mirantis.com> On Thu, Nov 26, 2015 at 3:23 AM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Stan, > > Thanks for looking into those files. > > They are what were generated on my machine. > > I tried one more time, and this time used a HOT package (something that > comes ootb) and noticed the ui.yaml file generated was still in the same > format as the previous ones. > I am attaching the full folder structure created for this HOT package > under /tmp/muranodashboard-cache/apps/ along with the hot package itself > that I imported. > > > > I'd appreciate it if you could take a look and let me know if you see > something wrong. Thanks. > > Regards, > --Vahid > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] 'NoMatchingFunctionException: No function "#operator_." matches supplied arguments' error when adding an application to an environment
Vahid,
The files you attached doesn't look like a YAML at all
Sincerely yours,
Stan Lagun
Principal Software Engineer @ Mirantis
<sla...@mirantis.com>
On Wed, Nov 25, 2015 at 10:21 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:
> Kate, Serg,
>
> Thank you very much for your quick responses.
>
> I am attaching the requested yaml file.
>
> For Hot2 plugin, the only difference with hot_package.py is that I changed
> the UI translation version to 2.2 and removed predefined_fields in
> _translate_ui_parameters method.
> The ui.yaml file is attached.
>
>
>
> The CSAR plugin is a work in progress. For example, the workflow method is
> not yet implemented.
> The ui.yaml file is attached.
>
>
> I'd like to make sure the issue is not on my side before I open a bug, as
> Kate suggested.
>
> If necessary, I can also send you the plugin python code.
>
> Thanks.
>
> Regards,
> --Vahid
>
>
>
>
> From:Serg Melikyan <smelik...@mirantis.com>
> To:"OpenStack Development Mailing List (not for usage questions)"
> <openstack-dev@lists.openstack.org>
> Date:11/25/2015 03:29 AM
> Subject:Re: [openstack-dev] [Murano]
> 'NoMatchingFunctionException: No function "#operator_." matches supplied
> arguments' error when adding an application to an environment
> --
>
>
>
> Hi Vahid,
>
> you can find generated UI definitions for the package here:
> /tmp/muranodashboard-cache/apps/.../ui/ui.yaml
>
> On Wed, Nov 25, 2015 at 12:54 PM, Ekaterina Chernova
> <efedor...@mirantis.com> wrote:
> >
> > Hi Vahid,
> >
> > Forms are rended after app is added to the environment in accordance
> with the UI definition.
> > UI definition contains yaql expressions and one of your expression is
> incorrect.
> > Please, share UI yaml so we can find out what's the problem.
> > You can also create a bug that it's not obvious which expression is
> failed.
> >
> >
> > Also, make sure that you have valid YAQL version installed (should
> correspond to the version in requirements.txt)
> > And you can ask for help in #murano channel.
> >
> > Regards,
> > Kate.
> >
> >
> > On Wed, Nov 25, 2015 at 4:39 AM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
> >>
> >> Hi,
> >>
> >> I am working on the TOSCA CSAR plugin for murano and so far am able to
> successfully import an application definition archive of my CSAR example to
> murano.
> >> However, when I try to add the imported application to an environment I
> get this error from Murano Dashboard:
> >>
> >> DEBUG:muranodashboard.common.cache:Using cached value from
> /tmp/muranodashboard-cache/apps/91/4c2bfd5d504419a94a9affb7af809a/package_fqn.
> >> DEBUG:muranodashboard.catalog.views:Clearing forms data for application
> io.murano.apps.generated.CsarHelloWorld.
> >> DEBUG:muranodashboard.catalog.views:Clearing any leftover wizard step
> data.
> >> DEBUG:muranodashboard.common.cache:Using cached value from
> /tmp/muranodashboard-cache/apps/91/4c2bfd5d504419a94a9affb7af809a/ui/ui.yaml.
> >> DEBUG:muranodashboard.common.cache:Using cached value from
> /tmp/muranodashboard-cache/apps/91/4c2bfd5d504419a94a9affb7af809a/package_fqn.
> >> DEBUG:muranodashboard.dynamic_ui.services:Using data {} for app
> io.murano.apps.generated.CsarHelloWorld
> >> DEBUG:muranodashboard.dynamic_ui.services:Using in-memory forms for app
> io.murano.apps.generated.CsarHelloWorld
> >> DEBUG:muranodashboard.common.cache:Using cached value from
> /tmp/muranodashboard-cache/apps/91/4c2bfd5d504419a94a9affb7af809a/ui/ui.yaml.
> >> DEBUG:muranodashboard.common.cache:Using cached value from
> /tmp/muranodashboard-cache/apps/91/4c2bfd5d504419a94a9affb7af809a/package_fqn.
> >> DEBUG:muranodashboard.dynamic_ui.services:Using data {} for app
> io.murano.apps.generated.CsarHelloWorld
> >> DEBUG:muranodashboard.dynamic_ui.services:Using in-memory forms for app
> io.murano.apps.generated.CsarHelloWorld
> >> INFO:muranodashboard.dynamic_ui.forms:Creating form workflowManagement
> >> INFO:muranodashboard.dynamic_ui.forms:Creating form group0
> >> DEBUG:muranodashboard.api:Murano::Client http://localhost:8082/>
> >> DEBUG:muranoclient.common.http:curl -i -X GET -H 'X-Auth-Token:
> 324759651d234c4eaf08f6093dfd7000' -H 'Content-Type: application/json' -H
> 'User-Agent: python-muranoclient'
> http://localhost:8082//v1/catalog/packages/914c2bfd5d504419a94a9affb7af809a
> >> DEBUG:muranoclient.common.http:
> >> HT
Re: [openstack-dev] [murano] Proposing Nikolai Starodubtsev for core
+1 Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis <sla...@mirantis.com> On Tue, Sep 1, 2015 at 3:03 PM, Alexander Tivelkov <ativel...@mirantis.com> wrote: > +1. Well deserved. > > -- > Regards, > Alexander Tivelkov > > On Tue, Sep 1, 2015 at 2:47 PM, Victor Ryzhenkin <vryzhen...@mirantis.com> > wrote: > >> +1 from me ;) >> >> -- >> Victor Ryzhenkin >> Junior QA Engeneer >> freerunner on #freenode >> >> Включено 1 сентября 2015 г. в 12:18:19, Ekaterina Chernova ( >> efedor...@mirantis.com) написал: >> >> +1 >> >> On Tue, Sep 1, 2015 at 10:03 AM, Dmitro Dovbii <ddov...@mirantis.com> >> wrote: >> >>> +1 >>> >>> 2015-09-01 2:24 GMT+03:00 Serg Melikyan <smelik...@mirantis.com>: >>> >>>> +1 >>>> >>>> On Mon, Aug 31, 2015 at 3:45 PM, Kirill Zaitsev <kzait...@mirantis.com> >>>> wrote: >>>> >>>>> I’m pleased to nominate Nikolai for Murano core. >>>>> >>>>> He’s been actively participating in development of murano during >>>>> liberty and is among top5 contributors during last 90 days. He’s also >>>>> leading the CloudFoundry integration initiative. >>>>> >>>>> Here are some useful links: >>>>> >>>>> Overall contribution: http://stackalytics.com/?user_id=starodubcevna >>>>> List of reviews: >>>>> https://review.openstack.org/#/q/reviewer:%22Nikolay+Starodubtsev%22,n,z >>>>> Murano contribution during latest 90 days >>>>> http://stackalytics.com/report/contribution/murano/90 >>>>> >>>>> Please vote with +1/-1 for approval/objections >>>>> >>>>> -- >>>>> Kirill Zaitsev >>>>> Murano team >>>>> Software Engineer >>>>> Mirantis, Inc >>>>> >>>>> >>>>> __ >>>>> OpenStack Development Mailing List (not for usage questions) >>>>> Unsubscribe: >>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> Serg Melikyan, Senior Software Engineer at Mirantis, Inc. >>>> http://mirantis.com | smelik...@mirantis.com >>>> >>>> +7 (495) 640-4904, 0261 >>>> +7 (903) 156-0836 >>>> >>>> >>>> __ >>>> OpenStack Development Mailing List (not for usage questions) >>>> Unsubscribe: >>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>>> >>> >>> >>> __ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >> __ >> >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [mistral][yaql] Addressing task result using YAQL function
I believe this is a good change. $.task_name requires you that $ be pointing to a tasks dictionary. But in the middle of the query like [1.2.3].select($ + 1) "$" will change its value. With a function approach you can write [1, 2, 3].select($ + task(taskName)). However the name "task" looks confusing as to my understanding tasks may have attributes other than result. It may make sense to use task(taskName).result instead. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis <sla...@mirantis.com> On Sun, Sep 6, 2015 at 12:14 AM, Dmitri Zimine <dzim...@stackstorm.com> wrote: > Yes meant to ask for consistency of referencing to task results. So it’s > task(task_name) regardless of where. > > One use case in favor of this is tooling: I refactor workflow with an > automated tool which wants to automatically rename the task name > EVERYWHERE. You guys know well by now that renaming the task is a source of > too many frustrating errors :) > > What other think? > > DZ. > > On Sep 3, 2015, at 4:23 AM, Renat Akhmerov <rakhme...@mirantis.com> wrote: > > > On 02 Sep 2015, at 21:01, Dmitri Zimine <dzim...@stackstorm.com> wrote: > > Agree, > > with one detail: make it explicit - task(task_name). > > > So do you suggest we just replace res() with task() and it looks like > > task() - get task result when we are in “publish” > task(task_name) - get task result from anywhere > > ? > > Is that correct you mean we must always specify a task name? The reason > I’d like to have a simplified form (w/o task name) is that I see a lot of > workflows that we have to repeat task name in publish so that it just look > too verbose to me. Especially in case of very long task name. > > Consider something like this: > > tasks: > *get_volumes_by_names*: > with-items: name in <% $.vol_names %> > workflow: get_volume_by_name name=<% $.name %> > publish: > volumes: <% $.*get_volumes_by_names* %> > > So in publish we have to repeat a task name, there’s no other way now. I’d > like to soften this requirement, but if you still want to use task names > you’ll be able to. > > > res - we often see folks confused by result of what (action, task, > workflow) although we cleaned up our lingo: action-output, task-result, > workflow-output…. but still worth being explicit. > > And full result is being thought as the root context $. > > Publishing to global context may be ok for now, IMO. > > > Not sure what you meant by "Publishing to global context”. Can you clarify > please? > > > Renat Akhmerov > @ Mirantis Inc. > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [murano] Questions on creating and importing HOT packages
1. This is for simplicity as file name doesn't really matter. HOT processing code needs to know where to look for template and it is easier to have fixed file name/location rather than force users to explicitly specify location in package manifest. But if if you have valid case where this becomes an issue please share it so that we could improve the code. BTW python-muranoclient may/should rename template file and what inside archive package is just an implementation detail 2. HOT syntax is not validated during import time as HOT validation is Heat's job and this can be done only during deployment time. However because Murano generates UI form and MuranoPL class syntax errors in HOT file may prevent it doing so. So this can be considered as some sort of validation but it is very very basic. Heat does much more in this area. We are doing this on server-side because this code is needed during deployment time (to generate MuranoPL on the fly) and also so that any changes made to HOT processor will immediately affect all HOT packages as this generation done on each deployment. This would be impossible if python-muranoclient would do the code generation (and this is the only thing that validates HOT in Murano). Also it is good to have clients thin to simplify development of clients for languages other then Python Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Sat, Aug 8, 2015 at 2:40 AM, Vahid S Hashemian vahidhashem...@us.ibm.com wrote: Hello, These two subjects have probably been discussed already, but I couldn't find any info on them, so very much appreciate if someone could clarify. 1. Why is the HOT template that is fed to package-create command renamed to template.yaml? Any issue with keeping the original name? 2. Why HOT syntax validation is deferred until package import time? Why not do it when creating the package? Thanks. Regards, - *Vahid Hashemian, Ph.D.* Advisory Software Engineer, IBM Cloud Labs __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [murano] [mistral] [yaql] Prepare to Yaql 1.0 release
Dmitry, this depends on how you're going to use yaql. yaql 1.0 has so
called legacy mode that is a layer on top of yaql that brings nearly full
backward compatibility including some of the things that were wrong in yaql
0.2. Use this mode when backward compatibility is a must. Without it the
following changes may affect queries written for 0.2:
1. There are no more tuples. foo(a = b) will mean now f(a='b') instead of
foo(tuple('a', 'b')) as it was in yaql 0.2. However dict(key1 = value1,
key2 = value2) and several other functions with the same syntax work
exactly how they used to work in 0.2
2. Contradictory to the first point there are no more lists. Yaql works on
immutable structures (tuple, frozenset and yaql1 own FrozenDict). All
operations that seem to modify data in fact return modified copies. All
input data will be automatically converted to their immutable versions and
converted back upon completion. This has 2 side effects: a) if you have
custom yaql functions they should follow the same pattern b) even if your
input data were build from tuples etc output will still use lists (e.g.
JSON format)
3. $dict.key will raise KeyError for missing keys instead of returning
None. Also key must be a valid keyword and cannot start with 2 underscores.
There are many other ways to retrieve dictionary value: $dict.get(value),
$dict.get(value, default), $dict[value], $dict[value, default]
4. In yaql 0.2 every function could be called as a method. So $x.foo() was
the same as foo($x). In yaql 1.0 it is up to function to decide if it wants
to be a function (foo($x), method ($x.foo()) or extension method (both
syntax).Considering that yaql handles different function overloads
$x.foo() may be a valid expression and execute something completely
different from foo($x).
Most of type-specific functions are declared as a methods. So you no
longer can say toUpper(str) but only str.toUpper()
5. yaql 0.2 had very few functions. yaql 1.0 has huge (~260
functions/operators etc) standard library. However several of the 0.2
functions now have different signature or name, replaced with better
alternative or just accept additional (optional) parameters
6. $collection[$ 0] doesn't work anymore. Use $collection.select($ 0)
for that
Sincerely yours,
Stan Lagun
Principal Software Engineer @ Mirantis
sla...@mirantis.com
On Tue, Aug 4, 2015 at 9:57 PM, Dmitri Zimine dzim...@stackstorm.com
wrote:
This is great news Alex, was looking forward to it, will be happy to
migrate Mistral.
Some heads-up on what syntactically changed would be much appreciated to
pass on to our users;
we likely will catch much of them with Mistral tests, but some may bubble
up.
DZ.
On Jul 27, 2015, at 2:04 AM, Alexander Tivelkov ativel...@mirantis.com
wrote:
Hi folks,
We are finally ready to release the 1.0.0 version of YAQL. It is a
huge milestone: the language finally looks the way we initially wanted
it to look. The engine got completely rewritten, tons of new
capabilities have been added. Here is a brief (and incomplete) list of
new features and improvements:
* Support for kwargs and keyword-only args (Py3)
* Optional function arguments
* Smart algorithm to find matching function overload without side effects
* Ability to organize functions into layers
* Configurable list of operators (left/right associative binary,
prefix/suffix unary with precedence)
* No global variables. There can be more than one parser with
different set of operators simultaneously
* List literals ([a, b])
* Dictionary literals ({ a = b})
* Handling of escape characters in string literals
* Verbatim strings (`...`) and double-quotes (...)
* =~ and !~ operators in default configuration (similar to Perl)
* - operator to pass context
* Alternate operator names (for example '*equal' instead of
'#operator_=')
so that it will be possible to have different symbol for particular
operator
without breaking standard library that expects operator to have well
known names
* Set operations
* Support for lists and dictionaries as a dictionary keys and set
elements
* New framework to decorate functions
* Ability to distinguish between functions and methods
* Switchable naming conventions
* Unicode support
* Execution options available to all invoked functions
* Iterators limitation
* Ability to limit memory consumption
* Can work with custom context classes
* It is possible to extend both parser and set of expression classes
on user-side
* It is possible to create user-defined types (also can be used for
dependency injection)
* Legacy yaql 0.2.x backward compatibility mode
* Comprehensive standard library of functions
* High unit test coverage
* Delegate and lambda support, including higher order lambdas
etc, etc.
So, this is a big change.
And as it always happens when moving from 0.something to 1.x the
breaking changes are inevitable. We have included the backwards
Re: [openstack-dev] [mistral] [murano] An online YAQL evaluator
Dmitry, yaql 1.0 has both str() and len() and much much more so there is no need to support them explicitly since Mistral is going to switch to yaql 1.0 and yaqluator.com is going to do the same Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Tue, Aug 4, 2015 at 8:55 PM, Dmitri Zimine dzim...@stackstorm.com wrote: Awesome! Really. Thank you folks for doing this. I am so much looking forward to moving it to 1.0 with more built-in functions and more power to extend it... Note that Mistral has a few extensions, like `str`, `len`, which are not in the scope of evaluator. DZ On Aug 2, 2015, at 12:44 PM, Stan Lagun sla...@mirantis.com wrote: Guys, this is awesome!!! Happy to see yaql gets attention. One more initiative that you may find interesting is https://review.openstack.org/#/c/159905/ This is an attempt to port yaql 1.0 from Python to JS so that the same can be done in browser Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Sun, Aug 2, 2015 at 5:30 PM, Nikolay Makhotkin nmakhot...@mirantis.com wrote: Hi guys! That's awesome! It is very useful for all us! -- Best Regards, Nikolay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org/?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [mistral] [murano] An online YAQL evaluator
Guys, this is awesome!!! Happy to see yaql gets attention. One more initiative that you may find interesting is https://review.openstack.org/#/c/159905/ This is an attempt to port yaql 1.0 from Python to JS so that the same can be done in browser Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Sun, Aug 2, 2015 at 5:30 PM, Nikolay Makhotkin nmakhot...@mirantis.com wrote: Hi guys! That's awesome! It is very useful for all us! -- Best Regards, Nikolay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral][Murano] What's the latest/greatest on YAQL?
The plan is to move to yaql 1.0 this release. Please do not merge yaql 1.0 into Mistral yet. Migration is going to happen really soon Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Fri, Jun 26, 2015 at 8:17 AM, Renat Akhmerov rakhme...@mirantis.com wrote: Yes, it’s a pretty important thing that we’d like to clarify as soon as possible. Any input from Murano team? Renat Akhmerov @ Mirantis Inc. On 26 Jun 2015, at 06:01, Dmitri Zimine dzim...@stackstorm.com wrote: Folks, it’s been some time,what’s the news: * Is Murano moving YAQL 1.0 in this cycle? * What’s your recommendation for this cycle - stay on 0.2.6 or move on? * Any progress on documentation? Thanks, DZ __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] Does Murano support nested HOT templates?
Currently Murano doesn't support nested template unless they are specified by URL. However this is going to be fixed in current release cycle as there are many requests for this feature. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Thu, Jun 11, 2015 at 1:55 AM, Vahid S Hashemian vahidhashem...@us.ibm.com wrote: Just wondering if Murano supports nested templates. For example, the autoscaling template at https://github.com/openstack/heat-templates/blob/master/hot/autoscaling.yaml references the template at https://github.com/openstack/heat-templates/blob/master/hot/lb_server.yaml. When I create the package, the referenced template does not seem to be included in the application archive. Even though the package is imported successfully, upon deployment an error is reported saying: ERROR: Failed to validate : : resources.asg: : Could not fetch remote template lb_server.yaml: Invalid URL scheme I'm not sure whether such templates should be imported differently or they are not supported yet. Regards, - Vahid Hashemian, Ph.D. Advisory Software Engineer, IBM Cloud Labs __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [murano] python versions
I'd rather go with Heat approach (job first) because it makes easier to track what is left to port to Py34 and track progress in this area Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Mon, Jun 8, 2015 at 2:46 PM, Kirill Zaitsev kzait...@mirantis.com wrote: I’ve looked into several OS projects, and they first seen to implement py3 support and create a job later. (Except for heat. They already have a non-voting py34, which seem to fail every time =)) I suggest we do the same: first make murano work on py34, then make a py34 job. I’ll file a blueprint shortly. -- Kirill Zaitsev Murano team Software Engineer Mirantis, Inc On 2 Jun 2015 at 15:58:17, Serg Melikyan (smelik...@mirantis.com) wrote: Hi Kirill, I agree with Alexander that we should not remove support for python 2.6 in python-muranoclient. Regarding adding python-3 jobs - great idea! But we need to migrate python-muranoclient to yaql 1.0 first and then add python-3 jobs, because previous versions of yaql are not compatible with python-3. On Tue, Jun 2, 2015 at 3:33 PM, Alexander Tivelkov ativel...@mirantis.com wrote: Hi Kirill, Client libraries usually have wider range of python requirements, as they may be run on various kinds of legacy environments, including the ones with python 2.6. only. Murano is definitely not the only project in Openstack which still maintains py26 compatibility for its client: nova, glance, cinder and other integrated ones do this. So, I would not drop py26 support for client code without any good reasons to. Are there any significant reasons to do it? Regarding py3.4 - this is definitely a good idea. -- Regards, Alexander Tivelkov On Tue, Jun 2, 2015 at 3:04 PM, Kirill Zaitsev kzait...@mirantis.com wrote: It seems that python-muranoclient is the last project from murano-official group, that still supports python2.6. Other projects do not have a 2.6 testing job (correct me if I’m wrong). Personally I think it’s time to drop support for 2.6 completely, and add (at least non-voting) jobs with python3.4 support tests. It seems to fit the whole process of moving OS projects towards python 3: https://etherpad.openstack.org/p/liberty-cross-project-python3 What do you think? Does anyone have any objections? -- Kirill Zaitsev Murano team Software Engineer Mirantis, Inc __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Serg Melikyan, Senior Software Engineer at Mirantis, Inc. http://mirantis.com | smelik...@mirantis.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] Nominating Kirill Zaitsev for murano-core
+1 without any doubt Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Tue, Jun 2, 2015 at 10:43 AM, Ekaterina Chernova efedor...@mirantis.com wrote: +1 Regards, Kate. On Tue, Jun 2, 2015 at 9:32 AM, Serg Melikyan smelik...@mirantis.com wrote: I'd like to propose Kirill Zaitsev to core members of Murano team. Kirill Zaitsev is active member of our community, he implemented https://launchpad.net/murano/+milestone/2015.1.0 several blueprint in Kilo and fixed number of bugs, he maintains a really good score as contributor: http://stackalytics.com/report/users/kzaitsev Existing Murano cores, please vote +1/-1 for the addition of Kirill to the murano-core. -- Serg Melikyan, Senior Software Engineer at Mirantis, Inc. http://mirantis.com | smelik...@mirantis.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] Nominating Filip Blaha for murano-core
+1 Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Tue, Jun 2, 2015 at 9:25 AM, Serg Melikyan smelik...@mirantis.com wrote: Folks, I'd like to propose Filip Blaha to core members of Murano team. Filip is active member of our community and he maintains a good score as contributor: http://stackalytics.com/report/users/filip-blaha Existing Murano cores, please vote +1/-1 for the addition of Filip to the murano-core. -- Serg Melikyan, Senior Software Engineer at Mirantis, Inc. http://mirantis.com | smelik...@mirantis.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
On Wed, May 13, 2015 at 5:42 PM, Zane Bitter zbit...@redhat.com wrote: There is more to multi-tenancy than just authentication/authorisation. It's also things like making sure one tenant's use of resources doesn't affect another tenant's (e.g. creating a denial of service by maxing out capacity); being able to measure and bill for usage; quotas to prevent accidental overuse c. This is a indeed a strong point that makes sense for shared MQ broker. However I'm not convinced that Zaqar approach is an optimal one: 1. There are a several dozens of message-oriented middleware solutions around (QPID, ActiveMQ, Camel, Kafka, HortnetQ and many more). Many of them are well know, well tested, have significant user base, published books etc. Does no one of them could be used instead with additional wrapping or special configs or additional plugins/extensions? So that is would left just to implement the integration layer instead of build MQ solution from scratch? 2. Even RabbitMQ has a lot of things pluggable and just maybe all those requirements can be met by a set of RabbitMQ plugins. It is also possible to contribute something to RabbitMQ or any other OSS MQ. 3. I'm not sure that noisy neighbor problem and other QoS related issues are a big problem here. The question is who is the consumer of MQ? If consumers are user-land applications running on OpenStack instance VMs a better solution could be just to set dedicated RabbitMQ/somethingMQ on a VM for those apps instead of a single shared server. I do understand that a common shared resource is easier to mange and administrate. On the other hand it is less secure and subject to QoS issues. There are so many software configuration management tools around (docker containers and container managers, prebuilt images, puppet manifests and so on) that it is not a real challenge any more. And if consumers are other OpenStack services QoS is unlikely to be a problem because in each OpenStack distribution there are fixed number of tested services with known shape of MQ workload. 4. Another option could be just to bring new MQ broker per tenant. For example to run docker RabbitMQ container per tenant + something small to manage them (to sync containers with tenant list). Docker has resource quotas that can be used. 5. Stable API is good. However open and standardized API like AMQP is even better. Maybe all of those possible solutions are bad and I'm sure there are downsides to each approach. It is just not obvious that development of yet another MQ solution worth it. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
On Wed, May 13, 2015 at 2:46 AM, Fox, Kevin M kevin@pnnl.gov wrote: Awesome. When is it/where do I go to look up that info? http://sched.co/3Clo Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
We are going to have a design session at the summit entirely dedicated to Murano guest agent security and isolation. Everyone is welcomed to attend and discuss security requirements, concerns and possible solutions. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Tue, May 12, 2015 at 8:32 PM, Fox, Kevin M kevin@pnnl.gov wrote: Barbican has the same issue. If Barbican is for storing secrets, how do you get a secret to the VM so it can get its secrets from Barbican? Aaaggg! :) I've been working on a solution here: https://blueprints.launchpad.net/barbican/+spec/vm-integration We're planning on talking more about that spec at the summit though. I've heard through the grape vine that Amazon lets you have a machine account that is associated with the vm. I'm not sure that's true or not, but some kind of keystone account integration into nova might help... Thanks, Kevin -- *From:* Georgy Okrokvertskhov [gokrokvertsk...@mirantis.com] *Sent:* Tuesday, May 12, 2015 10:06 AM *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [Murano] [Mistral] SSH workflow action There is one thing which still bothers me. It is authentication. Right now with separate RabbitMQ instance we keep VMs authentication isolated from OpenStack infra. This is still a problem if you want to use webhooks (Heat autoscaling, Murano actions) via our own authentication models. If we plan to use Zaqar it will be interesting to know how Zaqar solves this issue. Frankly, I don't think that this is a good idea to use Keystone credentials or tokens for MQ clients on VMs. This topic, probably, deserves its own e-mail thread. It will be interesting to discuss this with Keystone team. What is it is possible to have a token which is restricted to be authenticated to specific API URL like GET /v1/queues/queue-id/ Thanks Gosha On Tue, May 12, 2015 at 8:58 AM, Fox, Kevin M kevin@pnnl.gov wrote: +1 From: Zane Bitter [zbit...@redhat.com] Sent: Monday, May 11, 2015 6:15 PM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Murano] [Mistral] SSH workflow action Hello! This looks like a perfect soapbox from which to talk about my favourite issue ;) You're right about the ssh idea, for the reasons discussed related to networking and a few more that weren't (e.g. users shouldn't have to and generally don't want to give their private SSH keys to cloud services). I didn't know, or had forgotten, about the message queue implementation in Murano and while I think that's the correct shape for the solution, as long as the service in question is not multi-tenant capable it's a non-starter for a public clouds at least and probably many private clouds as well (after all, if you don't need multi-tenancy then why are you using OpenStack?). There's been a tendency within the application-facing OpenStack projects to hack together whatever local solutions to problems that we can in order to make progress without being held up by other projects. Let's take a moment to acknowledge that Heat is both the earliest and the biggest offender here, and that I am as culpable as anyone in the current state of affairs. There are multiple reasons for how things have gone - part of it is that it turned out we developed services in the wrong order, starting at too high a level. Part of it, frankly, is due to that element of the community that maintains a hostile position toward application-facing services and have used their influence in the community to maintain a disincentive against integrating projects together.[1] (If deployment of your project is discouraged that's one thing, but if it depends on another project whose deployment is also being discouraged then the hurdle you have to jump over is twice the height.) That said, I think we're at the point where we are hurting ourselves more than anyone else is by failing to come up with coherent, cross-project solutions. The problem articulated in this thread is not an isolated one. It's part of a more general pattern that affects a lot of projects: we need a way for the cloud to communicate to applications running in it. Angus started a recent discussion of this already on the list.[2] The requirements, IMHO, are roughly: * Reliability - we must be able to guarantee delivery to applications * Asynchrony - the cloud cannot block on user-controlled processes * Multitenancy - this is table stakes for OpenStack * Access control - even within tenants, we need to trust guest VMs minimally IMNSHO Zaqar messages are the obvious choice for the transport here. (Or something very similar in shape to Zaqar - but it'd be much better to join forces with the Zaqar team to improve it where necessary than to start a new project.) I really believe that if we
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
+1 for making Murano Engine - Murano Agent communication plugable so that one can switch to Zaqar or anything else. However watching RabbitMQ development for years I know hard can it be to build efficient and reliable system and I'm just not sure Zaqar can compete with such battle-proven thing like RabbitMQ yet. The only advantage I see is multi-tenancy. But I do believe it can be relatively easy be implemented with RabbitMQ. At lease in Murano. Don't want to go off topic here. The main idea is to use https://github.com/rabbitmq/rabbitmq-auth-backend-amqp and dynamically grant agent permissions only to his dedicated input queue so that it cannot access anything else. Not just other tenants queues but also queues of other VMs in the same tenant. In case of Murano we will not need to maintain additional secrets or databases. Neither it will be needed to create RabbitMQ users/vhosts as all of this becomes virtual. And agent will not be holding any RabbitMQ passwords at all Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Tue, May 12, 2015 at 10:52 AM, Renat Akhmerov rakhme...@mirantis.com wrote: Zane, Fully agree with you vision here. On 12 May 2015, at 07:15, Zane Bitter zbit...@redhat.com wrote: * Add an action in Mistral for sending a message to a Zaqar queue. This is easy and there's no reason you couldn't do it right now. Any volunteers? * Add a way to trigger a Mistral workflow with a Zaqar message. This is one piece in the puzzle to build user-configurable messaging flows between OpenStack services.[3] I added an agenda item for the summit in https://etherpad.openstack.org/p/vancouver-2015-design-summit-mistral to discuss this. Everyone is welcome. Imagine if there were one place where we implemented reliable queuing semantics at cloud scale, and when we added e.g. long-polling or WebSockets everyone could benefit immediately.[4] Imagine if there were one place for notifications, at cloud scale, for operators to secure. (How many webhook implementations are there in OpenStack right now? How many of them are actually secure against malicious users?) One format for messages between services so that users can connect up their own custom pipelines. We're not that far away! All of this is within reach if we work together. Cool picture of a wonderful future :) Thanks for reading. Please grab me at summit if you want to know more; I am always happy to bend the ear of anyone who will listen at length on this topic. As usual, I'll be the tall dude with the weird accent ;) With the great pleasure. (P.S. your accent is cool!) Renat Akhmerov @ Mirantis Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Filip, Currently there is no support in mistral how to execute scripts on VM via murano agent Mistral can call Murano application action that will do the job via agent. Actions are intended to be called by 3rd party systems with single HTTP request Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Mon, May 11, 2015 at 11:27 AM, Filip Blaha filip.bl...@hp.com wrote: Hi there is VPN mechanism in neutron we could consider for future how to get around these networking obstacles if we would like to use direct SSH. 1) every private created by murano would create VPN gateway on public interface of the router [1] neutron vpn-service-create --name myvpn --description My vpn service router1 mysubnet 2) any service like mistral which needs directly access VM via SSH (or other protocols) would connect to that VPN and then it could directly access VM on its fixed IP This mechanism would probably resolve network obstacles. But it requires more effort to analyse it. [1] https://wiki.openstack.org/wiki/Neutron/VPNaaS/HowToInstall Filip On 05/08/2015 10:22 AM, Renat Akhmerov wrote: Generally yes, std.ssh action works as long as network infrastructure allows access to a host using specified IP, it doesn’t provide anything on top of that. On 06 May 2015, at 22:26, Fox, Kevin M kevin@pnnl.gov kevin@pnnl.gov wrote: This would also probably be a good use case for Zaqar I think. Have a generic run shell commands from Zaqar queue agent, that pulls commands from a Zaqar queue, and executes it. The vm's don't have to be directly reachable from the network then. You just have to push messages into Zaqar. Yes, in Mistral it would be another action that puts a command into Zaqar queue. This type of action doesn’t exist yet but it can be plugged in easily. Should Mistral abstract away how to execute the action, leaving it up to Mistral how to get the action to the vm? Like I mentioned previously it should be just a different type of action: “zaqar.something” instead of “std.ssh”. Mistral engine itself works with all actions equally, they are just basically functions that we can plug in and use in Mistral workflow language. From this standpoint Mistral is already abstract enough. If that's the case, then ssh vs queue/agent is just a Mistral implementation detail? More precisely: implementation detail of Mistral action which may not be even hardcoded part of Mistral, we can rather plug them in (using stevedore underneath). Renat Akhmerov @ Mirantis Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribehttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Kevin, I do agree that lack of RabbitMQ multi-tenancy is a problem. However as this is developers mailing list I would suggest to contribute and make Murano guest agent secure for the benefit of all users and all existing applications instead of spending comparable amount of time developing intricate solutions that will make everything way more complex. If Murano application developers need to write additional Mistral workflow for each deployment step that would make application development be extremely harder and Murano be mostly useless. There are several approaches that can be taken to solve agent isolation problem and all of them are relatively easy to implement. This task is one of out top priorities for the Liberty and will be solved very soon anyway. Another approach that IMO also better than SSH is to use HOT software config that uses HTTP polling and doesn't suffer from lack of tenant isolation. I do want to see better Mistral integration in Murano as well as many other tools like puppet etc. And there are some good use cases for Mistral. But when it comes to the most basic things that Murano was designed to do from the foundation I want to make sure that Murano can do them the best way possible without requiring users to learn additional DSLs/tools or go extra step and involve additional services where not necessary. If something important is missing in Murano that makes usage of Mistral for deployment more attractive I'd rather focus on improving Murano and bringing those features to Murano. We can even use Mistral under the hood as long as we not make users to write both MuranoPL and Mistral DSL code for trivial things like service restart. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Sun, May 10, 2015 at 8:44 PM, Fox, Kevin M kevin@pnnl.gov wrote: Im planning on deploying murano but wont be supporting the murano guest agent. The lack of multi tenant security is a big problem I think. Thanks, Kevin -- *From:* Stan Lagun *Sent:* Saturday, May 09, 2015 7:21:17 PM *To:* OpenStack Development Mailing List (not for usage questions) *Subject:* Re: [openstack-dev] [Murano] [Mistral] SSH workflow action Filip, If I got you right the plan is to have Murano application execute Mistral workflow that SSH to VM and executes particular command? And alternative is Murano-Mistral-Zaquar-Zaquar agent? Why can't you just send this command directly from Murano (to Murano agent on VM)? This is the most common use case that is found in nearly all Murano applications and it is battle-proven. If you need SSH you can contribute SSH plugin to Murano (Mistral will require similar plugin anyway). The more moving parts you involve the more chances you have for everything to fail Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Fri, May 8, 2015 at 11:22 AM, Renat Akhmerov rakhme...@mirantis.com wrote: Generally yes, std.ssh action works as long as network infrastructure allows access to a host using specified IP, it doesn’t provide anything on top of that. On 06 May 2015, at 22:26, Fox, Kevin M kevin@pnnl.gov wrote: This would also probably be a good use case for Zaqar I think. Have a generic run shell commands from Zaqar queue agent, that pulls commands from a Zaqar queue, and executes it. The vm's don't have to be directly reachable from the network then. You just have to push messages into Zaqar. Yes, in Mistral it would be another action that puts a command into Zaqar queue. This type of action doesn’t exist yet but it can be plugged in easily. Should Mistral abstract away how to execute the action, leaving it up to Mistral how to get the action to the vm? Like I mentioned previously it should be just a different type of action: “zaqar.something” instead of “std.ssh”. Mistral engine itself works with all actions equally, they are just basically functions that we can plug in and use in Mistral workflow language. From this standpoint Mistral is already abstract enough. If that's the case, then ssh vs queue/agent is just a Mistral implementation detail? More precisely: implementation detail of Mistral action which may not be even hardcoded part of Mistral, we can rather plug them in (using stevedore underneath). Renat Akhmerov @ Mirantis Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] [Mistral] SSH workflow action
Filip, If I got you right the plan is to have Murano application execute Mistral workflow that SSH to VM and executes particular command? And alternative is Murano-Mistral-Zaquar-Zaquar agent? Why can't you just send this command directly from Murano (to Murano agent on VM)? This is the most common use case that is found in nearly all Murano applications and it is battle-proven. If you need SSH you can contribute SSH plugin to Murano (Mistral will require similar plugin anyway). The more moving parts you involve the more chances you have for everything to fail Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Fri, May 8, 2015 at 11:22 AM, Renat Akhmerov rakhme...@mirantis.com wrote: Generally yes, std.ssh action works as long as network infrastructure allows access to a host using specified IP, it doesn’t provide anything on top of that. On 06 May 2015, at 22:26, Fox, Kevin M kevin@pnnl.gov wrote: This would also probably be a good use case for Zaqar I think. Have a generic run shell commands from Zaqar queue agent, that pulls commands from a Zaqar queue, and executes it. The vm's don't have to be directly reachable from the network then. You just have to push messages into Zaqar. Yes, in Mistral it would be another action that puts a command into Zaqar queue. This type of action doesn’t exist yet but it can be plugged in easily. Should Mistral abstract away how to execute the action, leaving it up to Mistral how to get the action to the vm? Like I mentioned previously it should be just a different type of action: “zaqar.something” instead of “std.ssh”. Mistral engine itself works with all actions equally, they are just basically functions that we can plug in and use in Mistral workflow language. From this standpoint Mistral is already abstract enough. If that's the case, then ssh vs queue/agent is just a Mistral implementation detail? More precisely: implementation detail of Mistral action which may not be even hardcoded part of Mistral, we can rather plug them in (using stevedore underneath). Renat Akhmerov @ Mirantis Inc. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] python-openstackclient support
+1 for the idea though not sure on priority of this since we have so many way more important things to implement in Kilo. I'd say that would be a great contribution if we find someone willing to contribute it :) Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Thu, Apr 23, 2015 at 1:55 AM, Kirill Zaitsev kzait...@mirantis.com wrote: Since python-openstackclient is now a part of openstack — I guess it would be a good idea to support it in murano. It has setuptools-based plugin system, and it should be fairly easy to add murano commands as plugins to it. BTW, It’s based on cliff and has a terrific completion support (which is basically why I started looking into the issue in the first place =)) What do you think, is it a good idea? -- Kirill Zaitsev Sent with Airmail __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] [Murano] Mistral devstack installation is failing in murano gate job
1) yaql 1.0 is not a drop-in replacement for yaql 0.2 but only one version can be installed on any given system. Unless we use virtualenv, Docker or anything else to isolate applications. So if Murano and Mistral uses different yaql versions they will unable to live together on the same host 2) Currently we observe such impact on devstack tests but in general that will mean Murano and Mistral cannot be installed on the same DevStack or be together in some OpenStack distribution like Mirantis OpenStack at least the way they are deployed currently 3) yaql 1.0 is in beta status and is ready exactly to that degree. We don't expect any breaking changes anymore but it may still contain some bugs 4) Murano will move to yaql 1.0. We just didn't managed to do that in time before FF and it is too late to do that in Kilo 5) Generally we should have documentation for both versions. But the fact is that at the moment we don't even have documentation for 1.0 which is of a higher priority for us. So once again I suggest to contribute rather than wait for somebody else Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Mon, Apr 13, 2015 at 9:19 PM, Dmitri Zimine dzim...@stackstorm.com wrote: My 2c: Yes Mistral moved to YAQL 1.0 based on Murano team recommendations :) some questions/comments before we decide how to proceed: 1) Let’s clarify the impact: this problem doesn’t affect Murano directly; but it impacts Murano-Congress-Mistral initiative, correct? Is this a voting gate? What exactly is impacted? Are there any simpler workarounds? 2) on YAQL readiness: Mistral moved to YAQL it because 1) power 2) upcoming docs and 3) compatibility. We target to claim Mistral DSL “complete” in Kilo. YAQL is a big part of DSL from the user standpoint. Changing YAQL makes users migrate their workflows. Thus we want to stick to a version of YAQL which will be documented and used long term. If YAQL 1.0 is not ready in Kilo we should revert no questions. If it is ready, and comes with documentation - would it be good for Murano users if Murano moves to it? 3) given that YAQL 0.2 is supported for another cycle (.5 year) and users of both Mistral and Murano are using it, are there any plans to add documentation to it? It is the lack of docs on 0.2 is the biggest reason to push forward. (Does this sound like an invitation to cheat and offer no docs for 1.0 in kilo to convince Mistral to stay on 0.2?) DZ On Apr 13, 2015, at 6:13 AM, Serg Melikyan smelik...@mirantis.com wrote: Hi Nikolay Filip, indeed, root cause of the issue is that Murano Mistral use different version of yaql library. Murano installs yaql 0.2.4 and overrides 1.0.0b2 already installed and expected by Mistral. We decided that we are not going to switch to the yaql 1.0.0 in Kilo since we already finished Kilo development and working on bug-fixes and releasing RC. This gate may be fixed if only Mistral will revert 1.0.0 support in Kilo :'( Nikolay, what do you think about migrating to YAQL 1.0.0 in the next release? I know that it was me who proposed Mistral team to adopt yaql 1.0.0, and I am sorry, I didn't realize all consequences of moving Mistral to yaql 1.0.0 and Murano team living with yaql 0.2.4. We need to work on packaging and supporting yaql in Ubuntu/CentOS in order to add this library to the global-requirements and to avoid this kind of issues in the future. On Mon, Apr 13, 2015 at 3:58 PM, Nikolay Makhotkin nmakhot...@mirantis.com wrote: We are facing an issue with Mistral devstack installation in our gate job testing murano-congress-mistral integration (policy enforcement) [1] . Mistral devstack scripts are failing with following import error [2] Hi, Filip! Recently Mistral has moved to new YAQL, and it seems this dependency is missed (yaql 1.0, currently yaql 1.0.0b2) I think the root of problem is that Murano and Mistral have different yaql versions installed. -- Best Regards, Nikolay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Serg Melikyan, Senior Software Engineer at Mirantis, Inc. http://mirantis.com | smelik...@mirantis.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http
Re: [openstack-dev] [Mistral] Upgrading to YAQL 1.0
The plan was to integrate YAQL 1.0 into Murano so that we could say it is battle-proven, remove beta from version and release on PyPI. And then probably release v1.0.1 that is YAQL with documentation :) Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Mon, Apr 6, 2015 at 9:44 PM, Dmitri Zimine dzim...@stackstorm.com wrote: Renat, following up on the IRC meeting: I recalled one more item, and registered a blueprint - https://blueprints.launchpad.net/mistral/+spec/yaql-v1-0I think it’s good timing to do it in Kilo if YAQL 1.0 is announced; so I propose it for Kilo-rc1; One question is when YAQL 1.0 is going to move to Pypi? we can get it from github link for some time but ideally should use pypi. Alex, Stan, what’s the plan? DZ. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] Murano agent development
One trick that can be done to simplify agent development is to a) configure it to listen to some specific queue and run it locally or on VM (you can even run it from IDE under debugger) so that you can edit its code in place b) using code from agent.py write sample application that will send execution plan to you agent Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Tue, Mar 24, 2015 at 1:03 PM, Serg Melikyan smelik...@mirantis.com wrote: Hi Filip, Sure, murano-agent is just a python application, you can easily run this application locally (even in IDE like Pycharm with attached debugger). To send execution plan to the agent you need to publish execution plan to specified queue, results also will be published to the queue. murano-agent require configuration file for executing, you can build sample configuration file using following instruction: tox -e genconfig Please, take a look at example murano-agent configuration file that can be used to run it locally: [DEFAULT] # set location where obtained execution plans will be stored storage = /tmp/murano/plans debug = true verbose = true # credentials for rabbitmq [rabbitmq] host = localhost port = 5672 login = guest password = guest virtual_host = / # with this routing key execution plans results will be published result_routing_key = None # to this exchange execution plans results will be published result_exchange = None # from this queue agent will take exection plans input_queue = None On Tue, Mar 24, 2015 at 11:59 AM, Filip Blaha filip.bl...@hp.com wrote: Hello I would like to test and debug some new features of murano agent like chef recipes. I would like to know how to develop murano agent? How to test and debug new changes in agent code? Creating new image with every change and and test deployment on devstack is time-consuming and not very comfortable. Is there any shortcut for this development cycle? Thanks Filip __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Serg Melikyan, Senior Software Engineer at Mirantis, Inc. http://mirantis.com | smelik...@mirantis.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat][Mistral] Use and adoption of YAQL
I would suggest to do the migration but not to merge it till official yaql 1.0 release which is going to happen soon. As for the docs it is still very hard to write them since yaql 1.0 has got tons on new features and hundreds of functions. Any help is appreciated. But 99% of yaql 1.0 features and functions are covered by unit tests and there are ~250 of them so this is the best source of information at the moment. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Sat, Feb 7, 2015 at 12:46 AM, Dmitri Zimine dzim...@stackstorm.com wrote: Stan, Alex, Renat: Should we migrate to YAQL 1.0 now? and stop using the initial one? What’s the delta? Still short on the docs :) but I understand they’re coming up. https://github.com/stackforge/yaql/tree/master/doc/source Cheers, Dmitri. On Jan 16, 2015, at 6:46 AM, Stan Lagun sla...@mirantis.com wrote: Dmitri, we are working hard towards stable YAQL 1.0 which is expected to be released during Kilo cycle. It is going to have proper documentation and high unit test coverage which can also serve as a documentation source. YAQL has already migrated to StackForge and adopted OpenStack development process and tools but the work is still in progress. Any help from Mistral team and/or other YAQL adopters is appreciated. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Thu, Jan 15, 2015 at 10:54 PM, Dmitri Zimine dzim...@stackstorm.com wrote: Folks, We use YAQL in Mistral for referencing variables, expressing conditions, etc. Murano is using it extensively, I saw Heat folks thought of using it, at least once :) May be others... We are learning that YAQL incredibly powerful comparing to alternatives like Jinja2 templates used in salt / ansible. But with lack of documentation, it becomes one of adoption blockers to Mistral (we got very vocal user feedback on this). This is pretty much all the docs I can offer our users on YAQL so far. Not much. http://yaql.readthedocs.org/en/latest/ https://github.com/ativelkov/yaql/blob/master/README.rst https://murano.readthedocs.org/en/latest/murano_pl/murano_pl.html#yaql Are there any plans to fix it? Are there interest from other projects to use YAQL? Cheers, DZ. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://openstack-dev-requ...@lists.openstack.org/?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat][Mistral] Use and adoption of YAQL
Dmitri, we are working hard towards stable YAQL 1.0 which is expected to be released during Kilo cycle. It is going to have proper documentation and high unit test coverage which can also serve as a documentation source. YAQL has already migrated to StackForge and adopted OpenStack development process and tools but the work is still in progress. Any help from Mistral team and/or other YAQL adopters is appreciated. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Thu, Jan 15, 2015 at 10:54 PM, Dmitri Zimine dzim...@stackstorm.com wrote: Folks, We use YAQL in Mistral for referencing variables, expressing conditions, etc. Murano is using it extensively, I saw Heat folks thought of using it, at least once :) May be others... We are learning that YAQL incredibly powerful comparing to alternatives like Jinja2 templates used in salt / ansible. But with lack of documentation, it becomes one of adoption blockers to Mistral (we got very vocal user feedback on this). This is pretty much all the docs I can offer our users on YAQL so far. Not much. http://yaql.readthedocs.org/en/latest/ https://github.com/ativelkov/yaql/blob/master/README.rst https://murano.readthedocs.org/en/latest/murano_pl/murano_pl.html#yaql Are there any plans to fix it? Are there interest from other projects to use YAQL? Cheers, DZ. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] Nominating Kate Chernova for murano-core
+2 Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Thu, Dec 25, 2014 at 12:42 PM, Ruslan Kamaldinov rkamaldi...@mirantis.com wrote: Great addition to core team! +2 On Thu, Dec 25, 2014 at 11:02 AM, Timur Sufiev tsuf...@mirantis.com wrote: +1 from me. On Thu, Dec 25, 2014 at 10:28 AM, Serg Melikyan smelik...@mirantis.com wrote: I'd like to propose that we add Kate Chernova to the murano-core. Kate is active member of our community for more than a year, she is regular participant in our IRC meeting and maintains a good score as contributor: http://stackalytics.com/report/users/efedorova Please vote by replying to this thread. As a reminder of your options, +1 votes from 5 cores is sufficient; a -1 is a veto. -- Serg Melikyan, Senior Software Engineer at Mirantis, Inc. http://mirantis.com | smelik...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Timur Sufiev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] Murano Agent
Murano agent is required as long as you deploy applications that use it. You can take (write) application that uses Heat Software Configuration instead of Murano agent and use image without agent Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Mon, Dec 15, 2014 at 7:54 AM, Ruslan Kamaldinov rkamaldi...@mirantis.com wrote: On Mon, Dec 15, 2014 at 7:10 AM, raghavendra@accenture.com wrote: Please let me kow why Murano-agent is required and the components that needs to be installed in it. You can find more details about murano agent at: https://wiki.openstack.org/wiki/Murano/UnifiedAgent It can be installed with diskimage-builder: http://git.openstack.org/cgit/stackforge/murano-agent/tree/README.rst#n34 - Ruslan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] - puka dependency
Hi, It seems that you are using very old version of Murano because puka dependency was dropped several releases ago. Please see https://launchpad.net/murano for links to an up to date code and documentation Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Fri, Nov 21, 2014 at 9:38 AM, raghavendra@accenture.com wrote: Hi All, I am new to Murano trying to implement on Openstack Juno with Ubuntu 14.04 LTS version. However It needs puka 0.7 as dependency. I would appreciate any help with the build guides. I also find Github prompts for username and password while I try installing packages. Warm Regards, *Raghavendra Lad* -- This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. __ www.accenture.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
On Wed, Nov 12, 2014 at 3:50 PM, Zane Bitter zbit...@redhat.com wrote:
It's actually potentially horrible, because you introduce potential
quoting issues when you embed mistral workbooks in Heat templates or pass
Heat templates to Murano.
Could you please give an example of template with such issue?
Also note that
a. YAQL uses only single quote marks (') for string while JSON uses double
quotes ()
b. If Mistral worbook is embedded into HOT template it should not be parsed
or interpreted anyway. Otherwise consider what would happen if that
workbook would in turn have HOT snippets embedded
Sincerely yours,
Stan Lagun
Principal Software Engineer @ Mirantis
sla...@mirantis.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
My point is that quoting problem do exists probably but it exists even
without YAQL being used anywhere.
For example consider Mistral workbook containing value: { get_attr:
[my_instance, first_address] }. get_attr in Mistral may have nothing to to
with Heat's get_attr and even if it is it may be just a HOT snippet that
cannot be evaluated at the moment. For example it may be part of a HOT
template embedded into Mistral workbook and Mistral workflow is going to
create another Heat stack using that template. Now if you embed such
workbook into HOT template (and thus get HOT embedded into Mistral embedded
into HOT :) ) you need to make sure that Heat's parser is not going to try
to parse that workbook but treats it like a black box. This is regardless
of YAQL being even exist. But if you treat workbooks like
black-boxes/arbitrary JSON you should not care if it contains YAQL or not.
As for Murano this may cause some problems in cases when someone just took
YAML HOT and copy-paste it into MuranoPL class without double-checking. But
those errors can be easily fixed and such issues do exist even with current
HOT syntax without YAQL being used at all.
What I'm really confused by is why we have
a new weird ugly language like YAQL (sorry, it, like JQ, is hideous),
YAQL syntax is almost identical to Python's or JS if you narrow them to
one-line expressions. The only noticeable difference is that all
variable/parameter names are prefixed with '$' as in PHP/Perl which are
both popular languages. I can hardly imagine why such negligible difference
from JS or Python (languages you do like) makes language weird ugly
Sincerely yours,
Stan Lagun
Principal Software Engineer @ Mirantis
sla...@mirantis.com
On Wed, Nov 12, 2014 at 6:10 PM, Clint Byrum cl...@fewbar.com wrote:
Excerpts from Zane Bitter's message of 2014-11-11 13:06:17 -0800:
On 11/11/14 13:34, Ryan Brown wrote:
I am strongly against allowing arbitrary Javascript functions for
complexity reasons. It's already difficult enough to get meaningful
errors when you up your YAML syntax.
Agreed, and FWIW literally everyone that Clint has pitched the JS idea
to thought it was crazy ;)
So far nobody has stepped up to defend me, so I'll accept that maybe
people do think it is crazy. What I'm really confused by is why we have
a new weird ugly language like YAQL (sorry, it, like JQ, is hideous),
and that would somehow be less crazy than a well known mature language
that has always been meant for embedding such as javascript.
Anyway, I'd prefer YAQL over trying to get the intrinsic functions in
HOT just right. Users will want to do things we don't expect. I say, let
them, or large sections of the users will simply move on to something
else.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
For those who don't know 100% of guys behind YAQL are also active OpenStack contributors. During (early) Kilo cycle we plan to release YAQL version 1.0 on stackforge. This release is going to fix some flaws in early versions, add some more flexibility and have very high UT coverage. There are at least 2 projects in OpenStack that already make heavy use of YAQL: Mistral and Murano. If Heat team decides to use YAQL as well we will be glad to help integrate it smoothly with HOT as well as make sure that YAQL 1.0 is aligned with Heat use cases. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Wed, Nov 12, 2014 at 12:32 AM, Angus Salkeld asalk...@mirantis.com wrote: On Wed, Nov 12, 2014 at 1:35 AM, Alexis Lee alex...@hp.com wrote: Alexis Lee said on Mon, Nov 10, 2014 at 05:34:13PM +: How about we support YAQL expressions? https://github.com/ativelkov/yaql Plus some HOFs (higher-order functions) like cond, map, filter, foldleft etc? We could also use YAQL to provide the HOFs. Here's first_nonnull: config: Fn::Select - 0 filter: - yaql: $.0 != null - item1 - itemN config: yaql: $[$ != null][0] - item1 - itemN This approach requires less change to Heat, at the price of learning more YAQL. +1 to YAQL -Angus Alexis -- Nova Engineer, HP Cloud. AKA lealexis, lxsli. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Time to Samba! :-)
This can be addressed by Murano only if its deployed to the cloud (on VM belonging to some tenant). Having it on OpenStack service layer integrated with major OpenStack services sounds very promising. The problem I see is significant overlap with Keystone, especially in Kerberos and LDAP parts Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Sun, Aug 17, 2014 at 4:56 AM, Martinx - ジェームズ thiagocmarti...@gmail.com wrote: I know! :-P On 16 August 2014 21:17, Adam Lawson alaw...@aqorn.com wrote: Also, don't forget that AD != LDAP. ;) On Aug 16, 2014 5:16 PM, Adam Lawson alaw...@aqorn.com wrote: Doesn't Murano address this already? On Aug 16, 2014 2:35 PM, Martinx - ジェームズ thiagocmarti...@gmail.com wrote: I think that it would be great too! OpenLDAP-as-a-Service... With multi-domain support! :-) Nevertheless, last time I used Samba, was back in 2001... It is impressive these days! It worth take a look... I'm using it for about two months now, it is great! Cheers! On 16 August 2014 18:01, Clint Byrum cl...@fewbar.com wrote: Excerpts from Martinx - ジェームズ's message of 2014-08-16 12:03:20 -0700: Hey Stackers, I'm wondering here... Samba4 is pretty solid (up coming 4.2 rocks), I'm using it on a daily basis as an AD DC controller, for both Windows and Linux Instances! With replication, file system ACLs - cifs, built-in LDAP, dynamic DNS with Bind9 as a backend (no netbios) and etc... Pretty cool! In OpenStack ecosystem, there are awesome solutions like Trove, Solum, Designate and etc... Amazing times BTW! So, why not try to integrate Samba4, working as an AD DC, within OpenStack itself?! But, if we did that, what would be left for us to reinvent in our own slightly different way? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] The future of the integrated release
On Fri, Aug 15, 2014 at 7:17 PM, Sandy Walsh sandy.wa...@rackspace.com wrote: I recently suggested that the Ceilometer API (and integration tests) be separated from the implementation (two repos) so others might plug in a different implementation while maintaining compatibility, but that wasn't well received. Personally, I'd like to see that model extended for all OpenStack projects. Keep compatible at the API level and welcome competing implementations. Brilliant idea I'd vote for Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano]
Hi Steve, 1. There are no objections whatsoever if you know how to do it without breaking the entire concept 2. I thing that deployment workflow need to be broken to more fine-grained steps. Maybe instead of single deploy methdos have prepareDeploy (which doesn't push the changes to Heat), deploy and finishDeploy. Maybe more/other methods need to be defined. This will make the whole process more customizible 3. If you want to have single-instance applications based on a fixed prebuild image then maybe what you need is to have your apps inhertir both Application and Instance classes and then override Instance's deploy method and add HOT snippet before VM instantiation. This may also require ability for child class to bind fixed values to parent class properties (narrowing class public contract, hiding those properties from user). This is not yet supported in MuranoPL but can be done in UI form as a temporary workaround 4. Didn't get why you mentioned object model. Object model is mostly user input. Do you suggest passing HOT snippets as part of user input? If so that would be something I oppose to 5. I guess image tagging would be better solution to image-based deployment 6. Personally I believe that problem can be eficently solved by Murano today or in the nearest future without resorting to pure HOT packages. This is not against Murano design and perfectly alligned with it Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Tue, Jul 22, 2014 at 8:05 PM, McLellan, Steven steve.mclel...@hp.com wrote: Hi, This is a little rambling, so I’ll put this summary here and some discussion below. I would like to be able to add heat template fragments (primarily softwareconfig) to a template before an instance is created by Heat. This could be possible by updating but not pushing the heat template before instance.deploy, except that instance.deploy does a stack.push to configure networking before it adds information about the nova instance. This seems like the wrong place for the networking parts of the stack to be configured (maybe in the Environment before it tries to deploy applications). Thoughts? -- The long version: I’ve been looking at using disk-image-builder (a project that came out of triple-o) to build images for consumption through Murano. Disk images are built from a base OS plus a set of ‘elements’ which can include packages to install when building the image, templatized config file etc, and allows for substitutions based on heat metadata at deploy time. This uses a lot of the existing heat software config agents taking configuration from StructuredConfig and StructuredDeployment heat elements. I’m typically finding for our use cases that instances will tend to be single purpose (that is, the image will be created specifically to run a piece of software that requires some configuration). Currently Murano provisions the instance, and then adds software configuration as a separate stack-update step. This is quite inefficient since os-refresh-config ends up having to re-run, and so I’m wondering if there’s strong opposition to allowing the object model to support injection of software configuration heat elements before the instance is deployed. Alternatively maybe this is something that is best supported by pure HOT packages, but I think there’s value having murano’s composition ability even if just to be able to combine heat fragments (perhaps in the drag drop manner that was briefly discussed in Atlanta). Thanks, Steve ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] Image tagging
How do you like alternate design: uses can chose any image he wants (say
any Linux) but the JSON that is in image tag has enough information on what
applications are installed on that image. And not just installed or not but
the exact state installation was frozen (say binaries are deployed but
config files are need to be modified). The deployment workflow can peak
that state from image tag and continue right from the place it was stopped
last time. So if user has chosen image with MySQL preinstalled the workflow
will just post-configure it while if the user chosen clean Linux image it
will do the whole deployment from scratch. Thus it will become only a
matter of optimization and user will still be able to to share instance for
several applications (good example is Firewall app) or deploy his app even
if there is no image where it was built in.
Those are only my thoughts and this need a proper design. For now I agree
that we need to improve tagging to support yours use case. But this need to
be done in a way that would allow both user and machine to work with. UI at
least needs to distinguish between Linux and Windows while for user a
free-form tagging may be appropriate. Both can be stored in a single JSON
tag.
So lets create blueprint/etherpad for this and both think on exact format
that can be implemented right now
Sincerely yours,
Stan Lagun
Principal Software Engineer @ Mirantis
sla...@mirantis.com
On Tue, Jul 22, 2014 at 10:08 PM, McLellan, Steven steve.mclel...@hp.com
wrote:
Thanks for the response.
Primarily I’m thinking about a situation where I have an image that has a
specific piece of software installed (let’s say MySQL for the sake of
argument). My application (which configures mysql) requires a glance image
that has MySQL pre-installed, and doesn’t particularly care what OS (though
again for the sake of argument assume it’s linux of some kind, so that
configuration files are expected to be in the same place regardless of OS).
Currently we have a list of three hardcoded values in the UI, and none of
them apply properly. I’m suggesting instead of that list, we allow
free-form text; if you’re tagging glance images, you are expected to know
what applications will be looking for. This still leaves a problem in that
I can upload a package but I don’t necessarily have the ability to mark any
images as valid for it, but I think that can be a later evolution; for now,
I’m focusing on the situation where an admin is both uploading glance
images and murano packages.
As a slight side note, we do have the ability to filter image sizes based
on glance properties (RAM, cpus), but this is in the UI code, not enforced
at the contractual level. I agree reengineering some of this to be at the
contract level is a good goal, but it seems like that would involve major
reengineering of the dashboard to make it much dumber and go through the
murano API for everything (which ultimately is probably a good thing).
*From:* Stan Lagun [mailto:sla...@mirantis.com]
*Sent:* Sunday, July 20, 2014 5:42 AM
*To:* OpenStack Development Mailing List (not for usage questions)
*Subject:* Re: [openstack-dev] [Murano] Image tagging
Hi!
I think it would be usefull to share the original vision on tagging that
we had back in 0.4 era when it was introduced.
Tagging was supposed to be JSON image metadata with extendable scheme.
Workflow should be able to both utilize that metadata and impose some
constraints on it. That feature was never really designed so I cannot tell
exactly how this JSON should work or look like. As far as I see it it can
contain:
1. Operating system information. For example os: { family: Linux,
name: Ubuntu, version: 12.04, arch: x86_x64 } (this also may be
encoded as a single string)
Workflows (MuranoPL contracts) need to be able to express requirements
based on those attributes. For example
image:
Contract($.class(Image).check($.family = Linux and $.arch = x86)
In UI only those images that matches such contract should be displayed.
2. Human readable image title Ubuntu Linux 12.04 x86
3. Information about built-in software for image-based deployment. Not
sure exactly what information is needed. Meybe even portion of Object Model
so that if such image is used Murano environment will automatically
recocnize and incorporate that application like it was added by user to be
installed on clean instance. This will allow using of pre-build images with
preinstalled software (e.g. speed up deployment) but will make it
transparent for the user so that this software could be managed as well as
applications that user choses to install
4. Minimal hardware requirement for the image. Murano could use that
information to guarantee that user will not select flavor that is too small
for that operating system.
5. General-purposed tags
We need to think how this concept fits into our roadmap and new Glance
Re: [openstack-dev] [Murano] Image tagging
Hi!
I think it would be usefull to share the original vision on tagging that we
had back in 0.4 era when it was introduced.
Tagging was supposed to be JSON image metadata with extendable scheme.
Workflow should be able to both utilize that metadata and impose some
constraints on it. That feature was never really designed so I cannot tell
exactly how this JSON should work or look like. As far as I see it it can
contain:
1. Operating system information. For example os: { family: Linux,
name: Ubuntu, version: 12.04, arch: x86_x64 } (this also may be
encoded as a single string)
Workflows (MuranoPL contracts) need to be able to express requirements
based on those attributes. For example
image:
Contract($.class(Image).check($.family = Linux and $.arch = x86)
In UI only those images that matches such contract should be displayed.
2. Human readable image title Ubuntu Linux 12.04 x86
3. Information about built-in software for image-based deployment. Not sure
exactly what information is needed. Meybe even portion of Object Model so
that if such image is used Murano environment will automatically recocnize
and incorporate that application like it was added by user to be installed
on clean instance. This will allow using of pre-build images with
preinstalled software (e.g. speed up deployment) but will make it
transparent for the user so that this software could be managed as well as
applications that user choses to install
4. Minimal hardware requirement for the image. Murano could use that
information to guarantee that user will not select flavor that is too small
for that operating system.
5. General-purposed tags
We need to think how this concept fits into our roadmap and new Glance
design (probably there are other projects that can benefit from extended
image metadata) before chosing one of your approaches
Sincerely yours,
Stan Lagun
Principal Software Engineer @ Mirantis
sla...@mirantis.com
On Fri, Jul 18, 2014 at 6:46 PM, McLellan, Steven steve.mclel...@hp.com
wrote:
Hi,
Images that can be used for package deployment have to be tagged in glance
in order to enable the UI to filter the list of images to present to a user
(and potentially preselect). Currently the tags are defined in the
dashboard code (images/forms.py) which makes things very inflexible; if I
can upload an image and a package that consumes that image, I don’t want to
have to make a code change to use it.
Anyone who can upload images should also be able to specify tags for them.
There is also the question of whether a user should be allowed to tag
images that don’t belong to them (e.g. a shared image used by a private
package), but I think that can be punted down the road somewhat.
I think this needs to be more dynamic, and if that’s agreed upon, there
are a couple of approaches:
1) Store allowed tags in the database, and allow administrators to
add to that list. Ordinary users would likely not be able to create tags,
though they could use pre-defined ones for images they owned.
2) Have some public tags, but also allow user-specified tags for
private packages. I think this leads to all sorts of tricky edge cases
3) Allow freeform tags (i.e. don’t provide any hints). Since there’s
no formal link between the tag that a package looks for and the tags
currently defined in code, this wouldn’t make anything more susceptible to
inaccuracies
It would also be worth considering if there’s value allowing multiple tags
per image (I’m on the fence).
Personally, I think that 1) would be an improvement over the current
situation that’s reasonably easy to implement; that would allow the bare
minimum of flexibility without requiring much extra design. The other
options would perhaps be a longer term goal.
Thoughts? If there’s general agreement I will turn this into a blueprint.
Steve
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] Field 'name' is removed from Apps dynamic UI markup, should 'Version' be changed?
I we increment version to say 2.1 we could add code to dashboard to check for markup version and if it encounters version 2.0 to print verbose error telling how to migrate markup to 2.1. I don't see how both version can be supported simulteniously but at lease Version attribute must be checked and forms older version must fail with descriptive message rather than causing unpredictable behavior. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Fri, Jul 4, 2014 at 8:24 PM, Timur Sufiev tsuf...@mirantis.com wrote: Hi, folks! Recently we had decided to change a bit how Murano's dynamic UI works, namely do not explicitly specify 'name' field in first 'Add Application' form, but add it here automatically, since every component in Murano has a name. To avoid confusion with the 'name' field added by hand to the first form's markup, 'name' field on the first step will be forbidden and processing of an old UI markup which has such field will cause an exception. All these changes are described in the blueprint [1] in a greater detail. What is not entirely clear to me is whether should we increase 'Version' attribute of UI markup or not? On one hand, the format of UI markup is definitely changing - and old UI definitions won't work with the UI processor after [1] is implemented. It is quite reasonable to bump a format's version to reflect that fact. On the other hand, we will hardly support both format versions, instead we'll rewrite UI markup in all existing Murano Apps (there are not so many of them yet) and eventually forget that once upon a time the user needed to specify 'name' field explicitly. What do you think? [1] https://blueprints.launchpad.net/murano/+spec/dynamic-ui-specify-no-explicit-name-field -- Timur Sufiev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] Nominations to Murano core
+1 for both (or should I say +2?) Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Thu, Jun 26, 2014 at 1:49 PM, Alexander Tivelkov ativel...@mirantis.com wrote: +1 on both Serge and Steve -- Regards, Alexander Tivelkov On Thu, Jun 26, 2014 at 1:37 PM, Ruslan Kamaldinov rkamaldi...@mirantis.com wrote: I would like to nominate Serg Melikyan and Steve McLellan to Murano core. Serge has been a significant reviewer in the Icehouse and Juno release cycles. Steve has been providing consistent quality reviews and they continue to get more frequent and better over time. Thanks, Ruslan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] environment deletion
Hi Steve, The initial implementation of environment deletion (before 'destroy' method was introduced in MuranoPL) was to deploy environment with empty application list. So the code that deletes Heat stack was in 'deploy'. And it is still there. Environment's 'destroy' will be executed only in case Object Model is empty (doesn't contain Environment object), not in the case it is present, but application list is empty. Now if you say that stack doesn't get deleted it is clearly a bug that need to be filled in launchpad. I'm not sure why it can happen. Maybe API behavior was changed recently to send empty Object Model or it sends incorrect action name. This just need to be debugged. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Wed, Jun 25, 2014 at 3:57 AM, McLellan, Steven steve.mclel...@hp.com wrote: Is there any reason the system Environment class doesn’t implement destroy? Without it, the pieces of the heat stack not owned by other resources get left lying around. It looks like it was once implemented as part of deploy, but that no longer seems to execute. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] Murano API improvements
I think API need to be redesigned at some point. There is a blueprint for this: https://blueprints.launchpad.net/murano/+spec/api-vnext It seems reasonable to implement new API on new framework at once Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Mon, Jun 2, 2014 at 12:21 PM, Ruslan Kamaldinov rkamaldi...@mirantis.com wrote: Let's follow the standard procedure. Both blueprints lack specification of implementation details. There also has to be someone willing to implement these blueprints in near feature. I'm not opposed to these ideas and I'd really like to see Pecan added during Juno, but we still need to follow the procedure. I cannot approve an idea, it should be a specification. Let's work together on the new API specification first, then we'll need to find a volunteer to implement it on top of Pecan. -- Ruslan On Mon, Jun 2, 2014 at 8:35 AM, Timur Nurlygayanov tnurlygaya...@mirantis.com wrote: Hi all, We need to rewrite Murano API on new API framework and we have the commit: https://review.openstack.org/#/c/60787 (Sergey, sorry, but -1 from me, need to fix small isses) Also, today I created blueprint: https://blueprints.launchpad.net/murano/+spec/murano-api-workers this feature allows to run many API threads on one host and this allows to scale Murano API processes. I suggest to update and merge this commit with migration to Pecan framework and after that we can easily implement this blueprint and add many other improvements to Murano API and Murano python agent. Ruslan, could you please approve these blueprints and target them to some milestone? Thank you! -- Timur, QA Engineer OpenStack Projects Mirantis Inc ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] Application Actions
Agree. Lets for now assume nothing about events and have separate list of
blueprints for actions. As soon as we get back to event design we will
decide how they connected with actions and maybe then mark some of event
blueprints as obsolete/superseded
Sincerely yours,
Stan Lagun
Principal Software Engineer @ Mirantis
sla...@mirantis.com
On Fri, May 30, 2014 at 9:56 AM, Serg Melikyan smelik...@mirantis.com
wrote:
Hi, Alexander
Thank you for formalizing specifications and requirements! I will be glad
to work on implementation of app actions, once we finish discussing them. I
have created new blueprint
https://blueprints.launchpad.net/murano/+spec/application-actions and
referenced etherpad as specification.
Actually your specification in etherpad looks very comprehensive and quite
enough to implement first version of the feature. I think I can start
drafting implementation while we also working on specification at the same
time.
I think Events should be based on Actions, so we can return to Events in
the next milestone and revisit them.
On Thu, May 29, 2014 at 8:26 PM, Alexander Tivelkov
ativel...@mirantis.com wrote:
Hi folks!
During the Atlanta Summit there was quite a lot of talks about the
Application Lifecycle management and Murano's role in this process. There
were several cross-project sessions between Murano, Heat and Solum teams
([1]) at which it was decided that Murano has its own place in the
application-management ecosystem and should be able to define custom
actions or workflows for its applications, while using Heat and its ALM
capabilities as the underlying service.
At the same time I had some conversation with potential customers and
contributors, who have expressed strong interest in having actions in
Murano in this cycle.
That's why I've decided to drive this process forward and formalize the
spec and requirements for the Actions feature in Murano.
I've created a draft of the spec - please see the etherpad at [2] for
details. I'd like some comments and discussion on the spec, and once we all
agree on that, I will be happy to find a volunteer eager to implement this
during Juno :)
BTW, we have a number of blueprints already created on the topic ([3],
[4], [5], [6]), but they lack the details and have some problems with
terminology ('events' and 'actions' are definitely not the same, while the
blueprints mix them). I think we should revisit these BPs and either change
them to reflect the updated vision or to mark them as superseded and create
more appropriate one.
[1] https://etherpad.openstack.org/p/9XQ7Q2NQdv
[2] https://etherpad.openstack.org/p/MuranoActions
[3] https://blueprints.launchpad.net/murano/+spec/external-events
[4] https://blueprints.launchpad.net/murano/+spec/api-list-events
[5] https://blueprints.launchpad.net/murano/+spec/dsl-register-event
[6]
https://blueprints.launchpad.net/murano/+spec/ui-application-event-list
--
Regards,
Alexander Tivelkov
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Serg Melikyan, Senior Software Engineer at Mirantis, Inc.
http://mirantis.com | smelik...@mirantis.com
+7 (495) 640-4904, 0261
+7 (903) 156-0836
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] Improvements to MuranoPL contracts
Hi Alex,
1. This is directly related to the bug mentioned above.
Taking an example from launchpad
networks:
Contract:
environment: $.bool().notNull()
flat: $.bool().notNull()
custom: [$.class(Network).notNull()]
Default:
environment: true
flat: false
currently if no value was passed for 'networks' property (and no value
means no such key, null value doesn't count) then default value would be
used ({environment: true, flat: false}) and contract would be applied
to that value. If you pass networks: {flat: true} you will get
exception as this value doesn't matched contract (for 'environment' key).
With new approach exactly the same Default is interpreted as a set of
independent default - default for 'environment' attribute and default for
'flat' attribute. So {flat: true} turns into {environment: true,
flat: true}. With proposed changes the example above can be rewritten as
networks:
Contract:
environment: $.bool()
flat: $.bool()
custom: [$.class(Network)]
Default:
environment: true
2. Yes, I do agree it breaks compatibility. I believe we will brake it
anyway because MuranoPL is still far from mature and it is very likely that
some new future would require incompatible change. Thats was one of the
purpose of app-incubator repository - if we make breaking changes we can
walk through all applications and fix them. I hope that all such changes
will be made in Juno time frame.
As for this particular change it is 1% chance that $.int() contract was
intentionally written to accept null and 99% that it was mistake. So making
$.int() not-nullable we do much more good than cause harm
Sincerely yours,
Stan Lagun
Principal Software Engineer @ Mirantis
sla...@mirantis.com
On Tue, May 27, 2014 at 1:37 PM, Alexander Tivelkov
ativel...@mirantis.comwrote:
Hi Stan,
I don't understand well your third problem/solution statement. Could you
please give an example of the proposed behaviour vs the current one?
The last solution - to have not-nullable primitive types by default -
looks good to me, but it breaks the compatibility with the existing
contracts. If some package developers have already got $.int() in their
classes, we cannot be sure if this is a mistake or an intention to make the
property nullable. In the latter case their code will be broke, as the null
values will not be accepted anymore.
I understand that currently the adoption of MuranoPL is quite low and most
of its users are murano-contributors and so are probably reading this
mailing list, so this breaking change should not surprise them, but I'd
rather be careful with such things anyway. I suggest to discuss this today
on a weekly meeting in IRC. If everybody are fine with such changes, then
let's do them, as they are reasonable.
--
Regards,
Alexander Tivelkov
On Mon, May 26, 2014 at 6:40 PM, Stan Lagun sla...@mirantis.com wrote:
Hello everyone!
Recently I've been looking for a best way to fix incorrect handling of
defaults in MuranoPL's property contracts [1]. I analyzed how contracts
used in applications that are currently in app-incubator, typical mistakes
and usage patterns. I've found number of places where contract system can
be significantly improved. Here is my list of proposed improvements I'd
like to deliberate with you:
Problem: when contract violation happens it is very hard to tell without
debugger what went wrong. Especially this is a problem for complex nested
structures. Currently it even impossible to tell which property caused
exception during class load
Solution: during contract traversal keep track of path to a part of
contract being processed (for example 'foo/0/bar'). Include that path in
every thrown exception alongside with human-readable description describing
what value was processed and what contract was violated. Prepend property
name to exception text so that is would be clear what property cannot be
initialized. The same for method arguments.
---
Problem: Single default value is not enough for properties that are not
of scalar type.
For example if we have contract like
- name: $.string()
disabled: $.bool()
(array of structures) it is reasonable to have default value for
disabled attribute of each list entry whereas there is no meaningful
default for name attribute. Current approach allows to provide initial
filling of entire array which is obviously not what developer expects.
Solution: to have Default reflect contract structure so that different
parts of Default can serve as an independent defaults for corresponding
parts of the contract. Default need to be traversed in parallel with
contract spec and provided value. Default value need not provide value for
every single attribute mentioned in contract spec and thus can be simpler
than contract itself.
---
Problem: Default value is only used when no property value provided at
all. Null (None) value is a valid value even
Re: [openstack-dev] [Murano] Improvements to MuranoPL contracts
You're correct for not-nullable contract types. I agree that it is less
obvious. But on the other side it simplifies contract understanding because
otherwise developer needs to know not only how null is treated but also how
NoValue is treated. This also make UI development easier as with null equal
to NoValue you can have templated object model that you substitute values
from web form into. With nulls the structure always remain the same only
values change while with NoValue you have to delete key from template to
get default.
What is better? I don't know. Lets wait for other opinions
Sincerely yours,
Stan Lagun
Principal Software Engineer @ Mirantis
sla...@mirantis.com
On Tue, May 27, 2014 at 4:29 PM, Alexander Tivelkov
ativel...@mirantis.comwrote:
Got it.
What about this it is easier to set attribute value to None rather then
deleting attribute from object if default value is desired, especially for
generic clients?
Do I understand correctly that setting some property to None should leads
to automatic assignment of the default value?
As for me, this looks not obvious, and I would prefer not to have such
implicit assignments
--
Regards,
Alexander Tivelkov
On Tue, May 27, 2014 at 4:05 PM, Stan Lagun sla...@mirantis.com wrote:
Hi Alex,
1. This is directly related to the bug mentioned above.
Taking an example from launchpad
networks:
Contract:
environment: $.bool().notNull()
flat: $.bool().notNull()
custom: [$.class(Network).notNull()]
Default:
environment: true
flat: false
currently if no value was passed for 'networks' property (and no value
means no such key, null value doesn't count) then default value would be
used ({environment: true, flat: false}) and contract would be applied
to that value. If you pass networks: {flat: true} you will get
exception as this value doesn't matched contract (for 'environment' key).
With new approach exactly the same Default is interpreted as a set of
independent default - default for 'environment' attribute and default for
'flat' attribute. So {flat: true} turns into {environment: true,
flat: true}. With proposed changes the example above can be rewritten as
networks:
Contract:
environment: $.bool()
flat: $.bool()
custom: [$.class(Network)]
Default:
environment: true
2. Yes, I do agree it breaks compatibility. I believe we will brake it
anyway because MuranoPL is still far from mature and it is very likely that
some new future would require incompatible change. Thats was one of the
purpose of app-incubator repository - if we make breaking changes we can
walk through all applications and fix them. I hope that all such changes
will be made in Juno time frame.
As for this particular change it is 1% chance that $.int() contract was
intentionally written to accept null and 99% that it was mistake. So making
$.int() not-nullable we do much more good than cause harm
Sincerely yours,
Stan Lagun
Principal Software Engineer @ Mirantis
sla...@mirantis.com
On Tue, May 27, 2014 at 1:37 PM, Alexander Tivelkov
ativel...@mirantis.com wrote:
Hi Stan,
I don't understand well your third problem/solution statement. Could you
please give an example of the proposed behaviour vs the current one?
The last solution - to have not-nullable primitive types by default -
looks good to me, but it breaks the compatibility with the existing
contracts. If some package developers have already got $.int() in their
classes, we cannot be sure if this is a mistake or an intention to make the
property nullable. In the latter case their code will be broke, as the null
values will not be accepted anymore.
I understand that currently the adoption of MuranoPL is quite low and
most of its users are murano-contributors and so are probably reading this
mailing list, so this breaking change should not surprise them, but I'd
rather be careful with such things anyway. I suggest to discuss this today
on a weekly meeting in IRC. If everybody are fine with such changes, then
let's do them, as they are reasonable.
--
Regards,
Alexander Tivelkov
On Mon, May 26, 2014 at 6:40 PM, Stan Lagun sla...@mirantis.com wrote:
Hello everyone!
Recently I've been looking for a best way to fix incorrect handling of
defaults in MuranoPL's property contracts [1]. I analyzed how contracts
used in applications that are currently in app-incubator, typical mistakes
and usage patterns. I've found number of places where contract system can
be significantly improved. Here is my list of proposed improvements I'd
like to deliberate with you:
Problem: when contract violation happens it is very hard to tell
without debugger what went wrong. Especially this is a problem for complex
nested structures. Currently it even impossible to tell which property
caused exception during class load
Solution: during contract traversal keep track of path to a part
[openstack-dev] [Murano] Improvements to MuranoPL contracts
Hello everyone! Recently I've been looking for a best way to fix incorrect handling of defaults in MuranoPL's property contracts [1]. I analyzed how contracts used in applications that are currently in app-incubator, typical mistakes and usage patterns. I've found number of places where contract system can be significantly improved. Here is my list of proposed improvements I'd like to deliberate with you: Problem: when contract violation happens it is very hard to tell without debugger what went wrong. Especially this is a problem for complex nested structures. Currently it even impossible to tell which property caused exception during class load Solution: during contract traversal keep track of path to a part of contract being processed (for example 'foo/0/bar'). Include that path in every thrown exception alongside with human-readable description describing what value was processed and what contract was violated. Prepend property name to exception text so that is would be clear what property cannot be initialized. The same for method arguments. --- Problem: Single default value is not enough for properties that are not of scalar type. For example if we have contract like - name: $.string() disabled: $.bool() (array of structures) it is reasonable to have default value for disabled attribute of each list entry whereas there is no meaningful default for name attribute. Current approach allows to provide initial filling of entire array which is obviously not what developer expects. Solution: to have Default reflect contract structure so that different parts of Default can serve as an independent defaults for corresponding parts of the contract. Default need to be traversed in parallel with contract spec and provided value. Default value need not provide value for every single attribute mentioned in contract spec and thus can be simpler than contract itself. --- Problem: Default value is only used when no property value provided at all. Null (None) value is a valid value even if it not valid for particular contract. Thus is null is passed Default is not used. Solution: to treat every missing value as null. Missing Default is also considered to be null. This greatly simplifies understanding of contracts and client development (it is easier to set attribute value to None rather then deleting attribute from object if default value is desired, especially for generic clients) --- Problem: most of the time developer writes $.int() contract he doesn't realize that null is also valid value for such contract and the correct form is $.int().notNull(). This is even more obvious in case of bool(). Developers are lazy and usually forget to wright long verbose contracts. Solution: to make all primitive types be not-nullable with obvious defaults (0 for int, false for bool). This is enough for 95% of use cases. For the rest 5% where null value does valid to have separate contract methods optionalInt(), optionalBool() etc. As for strings I think that the same approach should be used - $.string() is a non-nullable sequence of chars with empty string as a default. And there should also be optionalString() that es equal to current $.string(). But in most cases when developer writes $.string() what he really wants is $.string().notNull().trim().notEmpty(). So while it is reasonable to have all this helper functions (notEmpty(), notBlank(), trim() etc.) is would be great to have one contract function that does exactly that. --- While some of proposed changes can possible break existing contracts in practice they just legalize current state of affairs so nothing would break. What do you think? [1]: https://bugs.launchpad.net/murano/+bug/1313694 Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [murano] Proposal to add Ruslan Kamaldinov to murano-core team
+1 Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com On Thu, Apr 17, 2014 at 6:51 PM, Georgy Okrokvertskhov gokrokvertsk...@mirantis.com wrote: +1 On Thu, Apr 17, 2014 at 6:01 AM, Dmitry Teselkin dtesel...@mirantis.comwrote: +1 Agree On Thu, Apr 17, 2014 at 4:51 PM, Alexander Tivelkov ativel...@mirantis.com wrote: +1 Totally agree -- Regards, Alexander Tivelkov On Thu, Apr 17, 2014 at 4:37 PM, Timur Sufiev tsuf...@mirantis.comwrote: Guys, Ruslan Kamaldinov has been doing a lot of things for Murano recently (including devstack integration, automation scripts, making Murano more compliant with OpenStack standards and doing many reviews). He's actively participating in our ML discussions as well. I suggest to add him to the core team. Murano folks, please say your +1/-1 word. -- Timur Sufiev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Thanks, Dmitry Teselkin Deployment Engineer Mirantis http://www.mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Georgy Okrokvertskhov Architect, OpenStack Platform Products, Mirantis http://www.mirantis.com Tel. +1 650 963 9828 Mob. +1 650 996 3284 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] [Murano] [Solum] applications in the cloud
On Fri, Apr 4, 2014 at 9:05 PM, Clint Byrum cl...@fewbar.com wrote: IMO that is not really true and trying to stick all these databases into one SQL database interface is not a use case I'm interested in pursuing. Indeed. Any SQL database is a useless interface. What I was trying to say is that some apps may work just on any MySQL while others require some specific version or variation or even impose some constraints on license or underlying operating system. One possible solution was to have some sort of interface hierarchy for that. Even better solution would be that all such properties be declared somewhere in HOT so that consumer could say not just I require MySQL-compatible template but I require MySQL-compatible template with version = 5.0 and clustered = True. Probably you can come with better example for this. Though interface alone is a good starting point. So for instance there is the non-Neutron LBaaS and the Neutron LBaaS, and both have their merits for operators, but are basically identical from an application standpoint. While conforming to the same interface from consumer's point of view different load-balancers has many configuration options (template parameters) and many of them are specific to particular implementation (otherwise all of them be functionally equal). This need to be addressed somehow. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
On Tue, Mar 25, 2014 at 2:27 PM, Thomas Herve thomas.he...@enovance.comwrote: What I can say is that I'm not convinced. The only use-case for a DSL would be if you have to upload user-written code, but what you mentioned is a Web interface, where the user doesn't use the DSL, and the cloud provider is the developer. There is no reason in this case to have a secure environment for the code. I didn't say that. There are at least 2 different roles application developers/publishers and application users. Application developer is not necessary cloud provider. The whole point of AppCatalog is to support scenario when anyone can create and package some application and that package can be uploaded by user alone. Think Apple AppStore or Google Play. Some cloud providers may configure ACLs so that user be allowed to consume applications they decided while others may permit to upload applications to some configurable scope (e.g. apps that would be visible to all cloud users, to particular tenant or be private to the user). We also think to have some of peer relations so that it would be possible to have application upload in one catalog to become automatically available in all connected catalogs. This is similar to how Linux software repos work - AppCatalog is repo, Murano package is what DEB/RPMs are to repo and DSL is what DEB/RPMs manifests are to packages. Just that is run on cloud and designed to handle complex multi-node apps as well as trivial ones in which case this may be narrowed to actual installation of DEB/RPM I'm glad that you bring packages up. This is a really good example of why you don't need a new programming language. Packages uses whatever technology they prefer to handle their scripting needs. They then have an declarative interface which hides the imperative parts behind. The same is true for Murano. MuranoPL is not used to express what should be deployed. In Murano there is object model that describes view of the word. It serves for the same purpose as HOT in Heat but it is simpler because it says just what need to be deployed but not how it should be accomplished as this information is already contained in application definitions. There is REST API to edit/submit Object Model which is again has nothing to do with MuranoPL. UI dashboard talks to AppCatalog to see what applications/classes are available and AppCatalog also knows what are the properties of those classes. This is needed for UI so that it can ask user for appropriate input. This is similar to how Horizon asks user to input parameters that are declared in HOT template. But all the imperative stuff is hidden inside Murano packages and is not used for anything outside Murano engine. MuranoPL is not a replacement for scripting languages. You still use bash/puppet/PowerShell/whatever you like for actual deployment. No MuranoPL code is executed on VM side. So the analogy with RPMs manifests is valid. You trust OpenStack developers with their code, you trust package developers with their code, why not trust catalog developers? They do trust catalog developers (hopefully). But catalog developers have nothing to do with catalog contents. Anyone can create and upload application to App Catalog the same way how anyone can upload his application to Google Play. The fact that I trust Google doesn't mean that I trust all applications in Google Play. The fact that I trust catalog developers doesn't mean that I (as a cloud operator) is going to allow execution of untrusted code in that catalog. Unless that code is sandboxed by design. Similar to that I can navigate to any web site google points me out and let it execute any JavaScript it wants unless it it JavaScript and not browser plugin or desktop application. -- Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Separating our Murano PL core in own package
I like dsl most because it is a. Short. This is especially good when you have that awesome 79-chars limitation b. It leaves a lot of room for changes. MuranoPL can change name. DSL - not :) On Mon, Mar 24, 2014 at 1:43 PM, Alexander Tivelkov ativel...@mirantis.comwrote: Hi Serg, Are you proposing to have a standalone git repository / stack forge project for that? Or just a separate package inside our primary murano repo? -- Regards, Alexander Tivelkov On Mon, Mar 24, 2014 at 12:00 PM, Serg Melikyan smelik...@mirantis.comwrote: Programming Language, AFAIK On Mon, Mar 24, 2014 at 11:46 AM, Oleg Gelbukh ogelb...@mirantis.comwrote: What does PL stand for, anyway? -- Best regards, Oleg Gelbukh On Mon, Mar 24, 2014 at 11:39 AM, Serg Melikyan smelik...@mirantis.comwrote: because 'dsl'/'language' terms are too broad. Too broad in general, but we choose name for sub-package, and in murano term 'language' mean Murano PL. +1 for language On Mon, Mar 24, 2014 at 11:26 AM, Timur Sufiev tsuf...@mirantis.comwrote: +1 for muranoapi.engine.murano_pl, because 'dsl'/'language' terms are too broad. On Mon, Mar 24, 2014 at 12:48 AM, Timur Nurlygayanov tnurlygaya...@mirantis.com wrote: Hi Serg, This idea sounds good, I suggest to use name 'murano.engine.murano_pl' (not just common name like 'language' or 'dsl', but name, which will be based on 'MuranoPL') Do we plan to support the ability to define different languages for Murano Engine? Thank you! On Sun, Mar 23, 2014 at 1:05 PM, Serg Melikyan smelik...@mirantis.com wrote: There is a idea to separate core of Murano PL implementation from engine specific code, like it was done in PoC. When this two things are separated in different packages, we will be able to track and maintain our language core as clean as possible from engine specific code. This will give to us an ability to easily separate our language implementation to a library. Questions is under what name we should place core of Murano PL? 1) muranoapi.engine.language; 2) muranoapi.engine.dsl; 3) suggestions? -- Serg Melikyan, Senior Software Engineer at Mirantis, Inc. http://mirantis.com | smelik...@mirantis.com +7 (495) 640-4904, 0261 +7 (903) 156-0836 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Timur, QA Engineer OpenStack Projects Mirantis Inc ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Timur Sufiev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Serg Melikyan, Senior Software Engineer at Mirantis, Inc. http://mirantis.com | smelik...@mirantis.com +7 (495) 640-4904, 0261 +7 (903) 156-0836 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Serg Melikyan, Senior Software Engineer at Mirantis, Inc. http://mirantis.com | smelik...@mirantis.com +7 (495) 640-4904, 0261 +7 (903) 156-0836 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] API/data formats version change and compatibility
Let's discuss this in community meeting. I would suggest drop support for older version at least until we release Murano 1.0. As soon as we start to guarantee backward compatibility we will introduce MinimalMuranoVersion instead of Version because format would not change but some particular template may require feature that was introduced after specified version. On Mon, Mar 24, 2014 at 3:42 PM, Timur Sufiev tsuf...@mirantis.com wrote: Hi all! While adapting Murano's Dynamic UI to the new MuranoPL data input format, I've encountered the need to add some new fields to it, which meant that the 'Version' field also had to be added to Dynamic UI. So, Dynamic UI definition without Version field is supposed to be of v.1 while upcoming Murano 0.5 release will support Dynamic UI v.2 processing (which produces data suitable for the MuranoPL in 0.5). But then the next questions arised. What if Murano 0.5 Dynamic UI processor gets definition in v.1 format? Should it be able to process it as well? If it should, then to which murano-api endpoint should it pass the resulting object model? I suspect that we have to deal with a slightly broader scope: to which extent should Murano components of each new version support the data formats and APIs from the previous versions? I suggest to discuss this question on the upcoming Murano's community meeting. -- Timur Sufiev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
On Mon, Mar 24, 2014 at 4:13 PM, Thomas Herve thomas.he...@enovance.comwrote: What I can say is that I'm not convinced. The only use-case for a DSL would be if you have to upload user-written code, but what you mentioned is a Web interface, where the user doesn't use the DSL, and the cloud provider is the developer. There is no reason in this case to have a secure environment for the code. I didn't say that. There are at least 2 different roles application developers/publishers and application users. Application developer is not necessary cloud provider. The whole point of AppCatalog is to support scenario when anyone can create and package some application and that package can be uploaded by user alone. Think Apple AppStore or Google Play. Some cloud providers may configure ACLs so that user be allowed to consume applications they decided while others may permit to upload applications to some configurable scope (e.g. apps that would be visible to all cloud users, to particular tenant or be private to the user). We also think to have some of peer relations so that it would be possible to have application upload in one catalog to become automatically available in all connected catalogs. This is similar to how Linux software repos work - AppCatalog is repo, Murano package is what DEB/RPMs are to repo and DSL is what DEB/RPMs manifests are to packages. Just that is run on cloud and designed to handle complex multi-node apps as well as trivial ones in which case this may be narrowed to actual installation of DEB/RPM -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral][TaskFlow] Long running actions
Don't forget HA issues. Mistral can be restarted at any moment and need to be able to proceed from the place it was interrupted on another instance. In theory it can be addressed by TaskFlow but I'm not sure it can be done without complete redesign of it On Fri, Mar 21, 2014 at 8:33 AM, W Chan m4d.co...@gmail.com wrote: Can the long running task be handled by putting the target task in the workflow in a persisted state until either an event triggers it or timeout occurs? An event (human approval or trigger from an external system) sent to the transport will rejuvenate the task. The timeout is configurable by the end user up to a certain time limit set by the mistral admin. Based on the TaskFlow examples, it seems like the engine instance managing the workflow will be in memory until the flow is completed. Unless there's other options to schedule tasks in TaskFlow, if we have too many of these workflows with long running tasks, seems like it'll become a memory issue for mistral... On Thu, Mar 20, 2014 at 3:07 PM, Dmitri Zimine d...@stackstorm.com wrote: For the 'asynchronous manner' discussion see http://tinyurl.com/n3v9lt8; I'm still not sure why u would want to make is_sync/is_async a primitive concept in a workflow system, shouldn't this be only up to the entity running the workflow to decide? Why is a task allowed to be sync/async, that has major side-effects for state-persistence, resumption (and to me is a incorrect abstraction to provide) and general workflow execution control, I'd be very careful with this (which is why I am hesitant to add it without much much more discussion). Let's remove the confusion caused by async. All tasks [may] run async from the engine standpoint, agreed. Long running tasks - that's it. Examples: wait_5_days, run_hadoop_job, take_human_input. The Task doesn't do the job: it delegates to an external system. The flow execution needs to wait (5 days passed, hadoob job finished with data x, user inputs y), and than continue with the received results. The requirement is to survive a restart of any WF component without loosing the state of the long running operation. Does TaskFlow already have a way to do it? Or ongoing ideas, considerations? If yes let's review. Else let's brainstorm together. I agree, that has major side-effects for state-persistence, resumption (and to me is a incorrect abstraction to provide) and general workflow execution control, I'd be very careful with this But these requirement comes from customers' use cases: wait_5_day - lifecycle management workflow, long running external system - Murano requirements, user input - workflow for operation automations with control gate checks, provisions which require 'approval' steps, etc. DZ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
and not consume it directly. So you still write code in
Turing-complete sophisticated language Java
2. Our initial idea was to translate MuranoPL into Mistral workflows
(similar to compiling C++ to assembly) but it turns to be too difficult to
implement
3. The DSL Mistral currently has is very very limited and academic. You
have to be real genius with a twisted mind to write real-life workflows
with Mistral DSL.
4. Mistral is at PoC stage and it would take him years to grow to something
that is as powerful as MuranoPL (or maybe BPEL)
5. Mistral does not need all that OOP stuff, inheritance, constraints etc.
And in practice it is very hard to separate declarations from the workflow
code as they are very coupled in their nature
6. DSL is a hart and brain of Murano. It is the most vital part of it. You
cannot develop system of such level as Murano without having control over
it. Any decision taken by Mistral team can become fatal for Murano.
Anyway I hope that some day we will be able to somehow merge our
DSLs/codebase/whatever
On Fri, Mar 21, 2014 at 10:20 PM, Zane Bitter zbit...@redhat.com wrote:
I completely agree with Georgy, but you raised some questions about Heat
that I want to answer in the interests of spreading knowledge about how
Heat works. A heavily-snipped response follows...
On 21/03/14 05:11, Stan Lagun wrote:
3. Despite HOT being more secure on the surface it is not necessary so
in reality. There is a Python class behind each entry in resources
section of HOT template. That Python code is run with root privileges
and not guaranteed to be safe. People make mistakes, forget to validate
parameters, make incorrect assumptions etc. Even if the code is proven
to be secure every single commit can introduce security breach. And no
testing system can detect this.
Quite right, I should acknowledge that it would be crazy to assume that
HOT is secure just because it is not a programming language, and I do not
make that assumption. (Indeed, YAML itself has been the subject of many
security problems, though afaik not in the safe mode that we use in Heat.)
Thanks for pointing out that I was not clear.
The operator can install whatever plugins they want.
They do but that is a bad solution. The reason is that plugins can
introduce additional resource types but they cannot modify existing
code. Most of the time cloud operators need to customize existing
resources' logic for their needs rather then rewriting it from scratch.
And they want their changes to be opaque to end-users. Imagine that
cloud operator need thats to get permission from his proprietary quota
management system for each VM spawned. If he would create custom
MyInstance resource type end-users could bypass it by using standard
Instance resource rather than custom one. Patching existing Python code
is not good in that then operator need to maintain his private fork of
the Heat and have troubles with CD, upgrades to newer versions etc.
It's not as bad as you think. All of the things you mentioned were
explicit design goals of the plugin system. If you install a plug-in
resource with the same type as a built-in resource then it replaces the
built-in one. And of course you can inherit from the existing plugin to
customise it.
So in this example, the operator would create a plugin like this:
from heat.engine.resources import server
from my.package import do_my_proprietary_quota_thing
class MyServer(server.Server):
def handle_create(self):
do_my_proprietary_quota_thing()
return super(MyServer, self).handle_create()
def resource_mapping():
return {'OS::Nova::Server': MyServer}
and drop it in /usr/lib/heat. As you can see, this is a simple
customisation (10 lines of code), completely opaque to end users
(OS::Nova::Server is replaced), and highly unlikely to be broken by any
changes in Heat (we consider the public APIs of
heat.engine.resource.Resource as a contract with existing plugins that we
can't break, at least without a lot of notice).
(I'm ignoring here that if this is needed for _every_ server, it makes no
sense to do it in Heat, unless you don't expose the Nova API to users at
all.)
Besides plugin system is not secure because plugins run with the
privileges of Heat engine and while I may trust Heat developers
(community) but not necessary trust 3rd party proprietary plugin.
I'm not sure who 'I' means in this context? As an end-user, you have no
way of auditing what code your cloud provider is running in general.
What
if he wants that auto-scaling would be based on input from his
existing
Nagios infrastructure rather then Ceilometer?
This is supported already in autoscaling. Ceilometer just hits a URL
for an alarm, but you don't have to configure it this way. Anything
can hit the URL.
And this is a good example for our general approach - we provide
Re: [openstack-dev] [legal-discuss] [Marconi] Why is marconi a queue implementation vs a provisioning API?
Kurt, Your point is that NoSQL solution may be required for innovative project. And that is MongoDB. But what if come another amazing project that needs CouchDB, neo4j, Riak, (put your favorite NoSQL DB here)? It would be in the same position cause everyone would say hey, we already have NoSQL in OpenStack so you have to use MongoDB which is not fair. But is also obvious that OpenStack cannot demand cloud operators to maintain MySQL, MongoDB, CouchDB, neo4j etc in simultaneously I hate to say that (I happen to be MongoDB fan) but the only way we can introduce external dependencies to OpenStack is by making technology that would make possible the project to be responsible for deployment and maintenance of that dependency (DBMS) rather then cloud operator. It seems to me that the right way to introduce MongoDB is to invest in projects like TripleO, Fuel, Murano, Heat and Ironic On Thu, Mar 20, 2014 at 9:09 PM, Gil Yehuda gyeh...@yahoo-inc.com wrote: To be clear, Marconi does not incorporate any AGPL code itself; pymongo is Apache2 licensed. Understood, but here's the rub. Someone else is going to want to build on this (which it the point of this open source project). Whereas 'pymongo' is Apache licensed, since the copyright holder, MongoDB Inc. declared it as such, the authors of the other community drivers (for other language bindings and features of MongoDB, etc.) are also of releasing drivers under the Apache or BSD licenses too (thinking that's OK to do since no one is telling them otherwise). That community is unaware of their legal obligations when creating drivers to an AGPL database. Thus if one of those community drivers gets intertwined in a court case clarifying their license to be infringing on the AGPL terms, we've inadvertently impacted our community. This is a credible risk that is difficult for OpenStack to abate, since the problem lies with the way a different community chose to operate. There are three interconnected issues here: 1. The confusion that MongoDB has created in Open Source licensing due to the asymmetric control they have on licensing terms. 2. The diligence of Open Stack to remain careful with OpenStack's CLA compliance and Apache-friendly terms. 3. The pragmatics of the effect MondgoDB would have onto OpenStack's economic viability and legal risks at large. The first problem is out of scope for this list. But I think people who rely upon Open Source for their business ought to understand what MongoDB is doing to open source software. The second is, to your point, the issue in this conversation. As long as Openstack only use Apache licensed code from MondgoDB Inc. and diligently avoids using any open source contributions from any community contributor to the MongoDB ecosystem, then you remain compliant the your CLA. But you will have to exclude the rest of the MongoDB community (which goes against the spirit of Open Source -- back to the problem #1, which is out of scope). As for #3, I think the foundation needs to weigh in on the pragmatics here, since this has an economic and legal impact to the entire endeavor, not just to persisting data in one component of the project. Gil Yehuda Sr. Director Of Open Source, Open Standards -Original Message- From: Kurt Griffiths [mailto:kurt.griffi...@rackspace.com] Sent: Thursday, March 20, 2014 9:29 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [legal-discuss] [Marconi] Why is marconi a queue implementation vs a provisioning API? The incorporation of AGPLv3 code Into OpenStack Project is a significant decision ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Marconi] Why is marconi a queue implementation vs a provisioning API?
as a non-AGPL alternative to MongoDB. For large-scale production deployments, we currently provide the MongoDB driver and will likely add Redis as another option (there is already a POC driver). And, of course, operators can provide drivers for NewSQL databases, such as VelocityDB, that are very fast and scale extremely well. In Marconi, every queue can be associated with a different backend cluster. This allows operators to scale both up and out, according to what is most cost-effective for them. Marconi's app-level sharding is currently done using a lookup table to provide for maximum operator control over placement, but I personally think it would be great to see this opened up so that we can swap in other types of drivers, such as one based on hash rings (TBD). 3. Cost-effective. The Marconi team has done a lot of work to (1) provide several dimensions for scaling deployments that can be used according to what is most cost-effective for a given use case, and (2) make the Marconi service as efficient as possible, including time spent optimizing the transport layer (using Falcon in lieu of Pecan, reducing the work that the request handlers do, etc.), and tuning the MongoDB storage driver (the SQLAlchemy driver is newer and we haven't had the chance to tune it yet, but are planning to do so during Juno). Turnaround on requests is in the low ms range (including dealing with HTTP), not the usec range, but that works perfectly well for a large class of applications. We've been benchmarking with Tsung for quite a while now, and we are working on making the raw data more accessible to folks outside our team. I'll try to get some of the latest data up on the wiki this week. Marconi was originally incubated because the community believed developers building their apps on top of OpenStack were looking for this kind of service, and it was a big missing gap in our portfolio. Since that time, the team has worked hard to fill that gap. Kurt ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
: Georgy, In consideration of the can you express it instead of the who will generate it, I see Heat's HOT evolving to support the expression of complex multi-tier architectures and applications (I would argue you can already do this today, perhaps with some additional features desired, e.g. Ability to define cloud workflows and workflow execution rules which could come when we have a workflow service like Mistral). Therefore, I would encourage Murano contributors to consider whether they can help make Heat sufficiently cover desired use cases. I have never viewed Heat templates as isolated components of a multi-tier architecture. Instead, a single template or a combination of master/subordinate templates together (using references, nesting, or inclusion) could express the complete architecture, both infrastructure and applications. If I've read your previous comments and threads correctly, you desire a way to express System Lifecycle Management across multiple related applications or components, whereby you view the System as a grouping of independently developed and/or deployed (but systematically related) components, whereby you view Components as individual disconnected Heat templates that independently describe different application stacks of the System. Did I get that correct? If so, perhaps the discussion here is one of scope of what can or should be expressed in a Heat template. Is it correct to state that your argument is that a separate system (such as Murano) should be used to express System Lifecycle Management as I've defined it here? If so, why could we not use the Heat DSL to also define the System? The System definition could be logically separated out into its own text file... But, we'd have a common DSL syntax and semantics for both lower level and higher level component interaction (a building block effect of sorts). As for who will generate it, ( with it being the Heat multi-tier application/infrastructure definition) I think that question will go through a lot more evolution and could be any number of sources: e.g. Solum, Murano, Horizon, Template Author with a text editor, etc. Basically, I'm a +1 for as few DSLs as possible. I support the position that we should evolve HOT if needed vs. having two separate DSLs that are both related to expressing application and infrastructure semantics. Workflow is quite interesting ... Should we be able to express imperative workflow semantics in HOT? Or, should we only be able to declare workflow configurations that get configured in a service like Mistral whereby Mistral's execution of a workflow may need to invoke Heat hooks or Stack Updates? Or, some other solution? I look forward to a design discussion on all this at the summit... This is fun stuff to think about! -Keith From: Georgy Okrokvertskhov gokrokvertsk...@mirantis.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Tuesday, March 18, 2014 1:49 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Murano][Heat] MuranoPL questions? I see this in the following way - who will generate HOT template for my complex multi-tier applications when I have only templates for components? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [openstack][Mistral] Adding new core reviewers
+1 for both On Wed, Mar 19, 2014 at 3:35 PM, Renat Akhmerov rakhme...@mirantis.comwrote: Team, So far I've been just the only one core member of the team. I started feeling lonely :) Since the project team and the project itself has now grown (thanks to StackStorm and Intel) I think it's time to think about extending the core team. I would propose: - Nikolay Makhotkin (nmakhotkin at launchpad). He's been working on the project since almost the very beginning and made significant contribution (design, reviews, code). - Dmitri Zimine (i-dz at launchpad). Dmitri joined the project about 2 months ago. Since then he's made a series of important high-quality commits, a lot of valuable reviews and, IMO most importantly, he has a solid vision of the project in general (requirements, use cases, comparison to other technologies) and has a pro-active viewpoint in all our discussions. Thoughts? Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
Management as I've defined it here? If so, why could we not use the Heat DSL to also define the System? The System definition could be logically separated out into its own text file... But, we'd have a common DSL syntax and semantics for both lower level and higher level component interaction (a building block effect of sorts). As for who will generate it, ( with it being the Heat multi-tier application/infrastructure definition) I think that question will go through a lot more evolution and could be any number of sources: e.g. Solum, Murano, Horizon, Template Author with a text editor, etc. Basically, I'm a +1 for as few DSLs as possible. I support the position that we should evolve HOT if needed vs. having two separate DSLs that are both related to expressing application and infrastructure semantics. Workflow is quite interesting ... Should we be able to express imperative workflow semantics in HOT? Or, should we only be able to declare workflow configurations that get configured in a service like Mistral whereby Mistral's execution of a workflow may need to invoke Heat hooks or Stack Updates? Or, some other solution? I look forward to a design discussion on all this at the summit... This is fun stuff to think about! -Keith From: Georgy Okrokvertskhov gokrokvertsk...@mirantis.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Tuesday, March 18, 2014 1:49 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Murano][Heat] MuranoPL questions? I see this in the following way - who will generate HOT template for my complex multi-tier applications when I have only templates for components? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano][Heat] MuranoPL questions?
Ability to hook up to application deployment is indeed a good thing. And it can be done both in HOT and MuranoPL. And Mistral is a good candidate to handle those hooks. But this is not a replacement for MuranoPL but an addition to it. The problem with hooks is that you cannot hoot just into arbitrary place in deployment workflow. And the author of Python code may not expose the exact hook that you need. Hooks can work for logging purposes or triggering some additional workflows but are not good for customization of your workflow from outside. Hooked code may not have access to all engine's internal state and workflow context and have even less chances to modify it in a safe manner On Thu, Mar 20, 2014 at 1:21 AM, Georgy Okrokvertskhov gokrokvertsk...@mirantis.com wrote: Hi, I think notification mechanism proposed in Heat will work fine for integration with external workflows. The approach which uses workflows outside of Heat engine sounds consistent with our current approach in Murano. I am looking into new TOSCA yaml format and I also ask Mirantis management to consider joining OASIS. The decision is not made yet, but hopefully will be made on next week. We eager to jump onto TOSCA standard work and contribute plan related parts. Thanks Georgy On Wed, Mar 19, 2014 at 1:38 PM, Thomas Spatzier thomas.spatz...@de.ibm.com wrote: Excerpts from Zane Bitter's message on 19/03/2014 18:18:34: From: Zane Bitter zbit...@redhat.com To: openstack-dev@lists.openstack.org Date: 19/03/2014 18:21 Subject: Re: [openstack-dev] [Murano][Heat] MuranoPL questions? On 19/03/14 05:00, Stan Lagun wrote: Steven, Agree with your opinion on HOT expansion. I see that inclusion of imperative workflows and ALM would require major Heat redesign and probably would be impossible without loosing compatibility with previous HOT syntax. It would blur Heat mission, confuse current users and rise a lot of questions what should and what should not be in Heat. Thats why we chose to built a system on top of Heat rather then expending HOT. +1, I agree (as we have discussed before) that it would be a mistake to shoehorn workflow stuff into Heat. I do think we should implement the hooks I mentioned at the start of this thread to allow tighter integration between Heat and a workflow engine (i.e. Mistral). +1 on not putting workflow stuff into Heat. Rather let's come up with a nice way of Heat and a workflow service to work together. That could be done in two ways: (1) let Heat hand off to a workflow service for certains tasks or (2) let people define workflow tasks that can easily work on Heat deployed resources. Maybe both make sense, but right now I am more leaning towards (2). So building a system on top of Heat is good. Building it on top of Mistral as well would also be good, and that was part of the feedback from the TC. To me, building on top means building on top of the languages (which users will have to invest a lot of work in learning) as well, rather than having a completely different language and only using the underlying implementation(s). That all sounds logical to me and would keep things clean, i.e. keep the HOT language clean by not mixing it with imperative expression, and keep the Heat engine clean by not blowing it up to act as a workflow engine. When I think about the two aspects that are being brought up in this thread (declarative description of a desired state and workflows) my thinking is that much (and actually as much as possible) can be done declaratively the way Heat does it with HOT. Then for bigger lifecycle management there will be a need for additional workflows on top, because at some point it will be hard to express management logic declaratively in a topology model. Those additional flows on-top will have to be aware of the instance created from a declarative template (i.e. a Heat stack) because it needs to act on the respective resources to do something in addition. So when thinking about a domain specific workflow language, it should be possible to define tasks (in a template aware manner) like on resource XYZ of the template, do something, or update resource XYZ of the template with this state, then do this etc. At runtime this would resolve to the actual resource instances created from the resource templates. Making such constructs available to the workflow authors would make sense. Having a workflow service able to execute this via the right underlying APIs would be the execution part. I think from an instance API perspective, Heat already brings a lot for this with the stack model, so workflow tasks could be written to use the stack API to access instance information. Things like update of resources is also something that is already there. BTW, we have a similar concept (or are working on a refinement of it based on latest discussions) in TOSCA and call it the plan
Re: [openstack-dev] MuranoPL questions?
Joshua, Completely agree with you. We wouldn't be writing another language if we knew how any of existing languages can be used for this particular purpose. If anyone suggest such language and show how it can be used to solve those issues DSL was designed to solve we will consider dropping MuranoPL. np Surely DSL hasn't stood the test of time. It just hasn't had a chance yet. 100% of successful programming languages were in such position once. Anyway it is the best time to come with your suggestions. If you know how exactly DSL can be replaced or improved we would like you to share On Wed, Mar 12, 2014 at 2:05 AM, Joshua Harlow harlo...@yahoo-inc.comwrote: I guess I might be a bit biased to programming; so maybe I'm not the target audience. I'm not exactly against DSL's, I just think that DSL's need to be really really proven to become useful (in general this applies to any language that 'joe' comp-sci student can create). Its not that hard to just make one, but the real hard part is making one that people actually like and use and survives the test of time. That's why I think its just nicer to use languages that have stood the test of time already (if we can), creating a new DSL (muranoPL seems to be slightly more than a DSL imho) means creating a new language that has not stood the test of time (in terms of lifetime, battle tested, supported over years) so that's just the concern I have. Of course we have to accept innovation and I hope that the DSL/s makes it easier/simpler, I just tend to be a bit more pragmatic maybe in this area. Here's hoping for the best! :-) -Josh From: Renat Akhmerov rakhme...@mirantis.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Monday, March 10, 2014 at 8:36 PM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] MuranoPL questions? Although being a little bit verbose it makes a lot of sense to me. @Joshua, Even assuming Python could be sandboxed and whatever else that's needed to be able to use it as DSL (for something like Mistral, Murano or Heat) is done why do you think Python would be a better alternative for people who don't know neither these new DSLs nor Python itself. Especially, given the fact that Python has A LOT of things that they'd never use. I know many people who have been programming in Python for a while and they admit they don't know all the nuances of Python and actually use 30-40% of all of its capabilities. Even not in domain specific development. So narrowing a feature set that a language provides and limiting it to a certain domain vocabulary is what helps people solve tasks of that specific domain much easier and in the most expressive natural way. Without having to learn tons and tons of details that a general purpose language (GPL, hah :) ) provides (btw, the reason to write thick books). I agree with Stan, if you begin to use a technology you'll have to learn something anyway, be it TaskFlow API and principles or DSL. Well-designed DSL just encapsulates essential principles of a system it is used for. By learning DSL you're leaning the system itself, as simple as that. Renat Akhmerov @ Mirantis Inc. On 10 Mar 2014, at 05:35, Stan Lagun sla...@mirantis.com wrote: I'd be very interested in knowing the resource controls u plan to add. Memory, CPU... We haven't discussed it yet. Any suggestions are welcomed I'm still trying to figure out where something like https://github.com/istalker2/MuranoDsl/blob/master/meta/com.mirantis.murano.demoApp.DemoInstance/manifest.yamlwould be beneficial, why not just spend effort sand boxing lua, python... Instead of spending effort on creating a new language and then having to sandbox it as well... Especially if u picked languages that are made to be sandboxed from the start (not python)... 1. See my detailed answer in Mistral thread why haven't we used any of those languages. There are many reasons besides sandboxing. 2. You don't need to sandbox MuranoPL. Sandboxing is restricting some operations. In MuranoPL ALL operations (including operators in expressions, functions, methods etc.) are just those that you explicitly provided. So there is nothing to restrict. There are no builtins that throw AccessViolationError 3. Most of the value of MuranoPL comes not form the workflow code but from class declarations. In all OOP languages classes are just a convenient to organize your code. There are classes that represent real-life objects and classes that are nothing more than data-structures, DTOs etc. In Murano classes in MuranoPL are deployable entities like Heat resources application components, services etc. In dashboard UI user works with those entities. He (in UI!) creates instances of those classes, fills their property values, binds objects together
Re: [openstack-dev] [Mistral][Taskflow][all] Mistral + taskflow
Joshua, why wait? Why not just help Renat with his research on that integration and bring your own vision to the table? Write some 1-page architecture description on how Mistral can be built on top of TaskFlow and we discuss pros and cons. In would be much more productive. On Fri, Mar 14, 2014 at 11:35 AM, Joshua Harlow harlo...@yahoo-inc.comwrote: Thanks Renat, I'll keep waiting, and hoping that we can figure this out for everyone's benefit. Because in the end we are all much stronger working together and much weaker when not. Sent from my really tiny device... On Mar 13, 2014, at 11:41 PM, Renat Akhmerov rakhme...@mirantis.com wrote: Folks, Mistral and TaskFlow are significantly different technologies. With different set of capabilities, with different target audience. We may not be doing enough to clarify all the differences, I admit that. The challenge here is that people tend to judge having minimal amount of information about both things. As always, devil in the details. Stan is 100% right, seems is not an appropriate word here. Java seems to be similar to C++ at the first glance for those who have little or no knowledge about them. To be more consistent I won't be providing all the general considerations that I've been using so far (in etherpads, MLs, in personal discussions), it doesn't seem to be working well, at least not with everyone. So to make it better, like I said in that different thread: we're evaluating TaskFlow now and will share the results. Basically, it's what Boris said about what could and could not be implemented in TaskFlow. But since the very beginning of the project I never abandoned the idea of using TaskFlow some day when it's possible. So, again: Joshua, we hear you, we're working in that direction. I'm reminded of http://www.slideshare.net/RenatAkhmerov/mistral-hong-kong-unconference-trac k/2http://www.slideshare.net/RenatAkhmerov/mistral-hong-kong-unconference-track/2where it seemed like we were doing much better collaboration, what has happened to break this continuity? Not sure why you think something is broken. We just want to finish the pilot with all the 'must' things working in it. This is a plan. Then we can revisit and change absolutely everything. Remember, to the great extent this is research. Joshua, this is what we talked about and agreed on many times. I know you might be anxious about that given the fact it's taking more time than planned but our vision of the project has drastically evolved and gone far far beyond the initial Convection proposal. So the initial idea of POC is no longer relevant. Even though we finished the first version in December, we realized it wasn't something that should have been shared with the community since it lacked some essential things. Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral][Taskflow][all] Mistral + taskflow
Joshua, Boris, Renat, I call for this discussion to be technical rather than emotional. Seems is not appropriate word here. It seems like both Mistral and TaskFlow duplicate many similar services and libraries outside of OpenStack (BTW, what exactly in TaskFlow is OpenStack-specific?). So it is not a good idea to judge who duplicates who and focus on how TaskFlow can help Mistral (or vice versa) Let us first list what can be considered to be inalienable parts of Mistral (features and use cases that are vital to Mistral paradigm) and then suggest how those use cases can be addressed by TaskFlow. I think it would be best if Joshua propose detailed Mistral design based on TaskFlow that would have all the listed futures and addresses all principal use cases. After all it really doesn't matter if Mistral and TaskFlow has similar pieces of code in their implementation or similar concepts unless those pieces of code and concepts can be extracted from TaskFlow and reused in Mistral On Fri, Mar 14, 2014 at 3:17 AM, Boris Pavlovic bo...@pavlovic.me wrote: Joshua, Fully agree, seems like Mistral duplicates a lot of efforts of TaskFlow. I just don't see any reason why Mistral is reimplementing TaskFlow and not just adding new features to... Best regards, Boris Pavlovic On Fri, Mar 14, 2014 at 3:02 AM, Joshua Harlow harlo...@yahoo-inc.comwrote: Separating from the following: * http://lists.openstack.org/pipermail/openstack-dev/2014-March/029870.html What can we do to resolve this, and reduce duplication in effort, code and so on. I believe that we are really doing the same thing, but it appears that something (imho) is wrong with the process here. I'm unsure how to resolve the situationŠ I believe since as a community we should be working together instead of working apart in silos creating the same thing. I'm reminded of http://www.slideshare.net/RenatAkhmerov/mistral-hong-kong-unconference-trac k/2http://www.slideshare.net/RenatAkhmerov/mistral-hong-kong-unconference-track/2where it seemed like we were doing much better collaboration, what has happened to break this continuity? How can we fix it? -Josh ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Crack at a Real life workflow
On Mon, Mar 10, 2014 at 12:26 PM, Renat Akhmerov rakhme...@mirantis.comwrote: In case of Amazon SWF it works in the opposite way. First of all it's a language agnostic web service, then they have language specific frameworks working on top of the service. The big question here is would SWF be popular (or at least usable in real-world scenarios) without language-specific framework on top? -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Mistral] Crack at a Real life workflow
deserialized version of it. Thus you can control where those YAMLs are located. You can store them in Glance, database, file-system or generate them on the fly without DSL even notice. 9. With YAML you has less to explain (people already know it) and more of a tooling and IDE support Q: Isn't DSL of that power would be a subject to DoS attacks, resource starvation attacks etc.? A: This is a really good question cause nobody asked it yet :) And the answer is NO. Here is why: With MuranoPL you have complete control over everything. You can have time quota for DSL code (remember 30 second quota for PHP scrips in typical web setup?). Or you can limit DSL scripts to some reasonable number of executed instructions. Or count raw CPU time for the script. You can make all loop constructs to have upper limit on iterations. You can do the same with YAQL functions. You can enforce timeouts on all I/O operations. MuranoPL uses green threads and you can have a lot of them (unlike regular threads) and it is possible to limit total number of green threads used for DSL execution. It is possible to even limit memory usage. DSL code cannot allocate operating-system-level resources like file handles of TCP sockets. And there is garbage collector as DSL in interpreted on top of Python. How easy it would be to do the same with JavaScript/Lua/whatever? On Sat, Mar 8, 2014 at 7:05 AM, Joshua Harlow harlo...@yahoo-inc.comwrote: What are the benefits of MuranoPL over a already established language? I noted the following and don't quite understand why they are needed (reinventing a language): - https://wiki.openstack.org/wiki/Murano/DSL/Blueprint#Extends - https://wiki.openstack.org/wiki/Murano/DSL/Blueprint#Block_constructs Q: where are those looping constructs executed? How hard is it to DOS murano servers (submitting jobs that loop forever). What execution limits are imposed? I noted that the parallel construct actually exposes the number of green threads (isn't this an avenue for resource starvation?). - https://wiki.openstack.org/wiki/Murano/DSL/Blueprint#Object_model IMHO, something just doesn't seem right when the above is created, fitting a language into YAML seems about as awkward as creating a language in XML (xquery[1] for example) . Why was this really preferred over just python or something simpler for example, [lua, javascript...], that already has language/object constructs... built-in and have runtimes that u can control the security domain of (python is not a good choice to run arbitrary code-in, find out how much energy google put into python + app-engine and u'll see what it takes). http://en.wikipedia.org/wiki/XQuery#Examples From: Stan Lagun sla...@mirantis.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Friday, March 7, 2014 at 9:36 AM To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Mistral] Crack at a Real life workflow Hello everyone! Actually it is possible to construct YAML-based DSL that has all the constructs of regular OOP language like Python and at the same time be safe enough to be used for execution of untrusted code on shared server. Take a look at Murano DSL. For example the code above defines class Instance: https://github.com/istalker2/MuranoDsl/blob/master/meta/com.mirantis.murano.services.Instance/manifest.yaml The part that may be useful for Mistral is under Workflow key. Here is some doc on the language: https://wiki.openstack.org/wiki/Murano/DSL/Blueprint Technically you can code any workflow that you can in Python using such language (just don't look at all OOP-related stuff) and it will look very similar to Python but be safe as you can only call APIs that are explicitly provided for DSL Hope this might be helpful for Mistral On Fri, Mar 7, 2014 at 10:38 AM, Dmitri Zimine d...@stackstorm.com wrote: I just moved the sample to Git; let's leverage git review for specific comments on the syntax. https://github.com/dzimine/mistral-workflows/commit/d8c4a8c845e9ca49f6ea94362cef60489f2a46a3 DZ On Mar 6, 2014, at 10:36 PM, Dmitri Zimine d...@stackstorm.com wrote: Folks, thanks for the input! @Joe: Hopefully Renat covered the differences. Yet I am interested in how the same workflow can be expressed as Salt state(s) or Ansible playbooks. Can you (or someone else who knows them well) take a stub? @Joshua I am still new to Mistral and learning, but I think it _is_ relevant to taskflow. Should we meet, and you help me catch up? Thanks! @Sandy: Aaahr, I used the D word?! :) I keep on arguing that YAML workflow representation doesn't make DSL. And YES to the object model first to define the workflow, with YAML/JSON/PythonDSL/what-else as a syntax to build it. We are having these discussions on another thread and reviews. Basically
Re: [openstack-dev] [Mistral] Crack at a Real life workflow
I would really like to answer all your questions but lets do it in separate thread :) It will just be a tight rope walk to keep that DSL free of issues when it enables all those features. MuranoPL doesn't built on Python - MuranoPL engine does. You can re-implement it on Java, Haskel whatever without loosing compatibility. So you never get (and never will) access to Python built-ins, Python modules and implementation details. DSL is safe to the level you want. All functions, operators, classes etc. that are available DSL are those that are written in MuranoPL or explicitly injected into DSL. So this is completely sandboxed. And in Murano we are not going to provide any operating-system level API. No file access, no direct networking. Just APIs to talk to other OpenStack services similar to their native python-clients. So there is no real need to physically isolate MuranoPL executions. As for your questions on pointers to execution control and resource constraints currently we do not have them. We are on PoC stage currently. There is a PoC MuranoPL engine that can execute most of the things but it lucks some advanced features like those you're talking about. But DSL engine was designed with such features in mind and they will be implemented during next several release cycles. It relatively easy to limit execution time, number of green threads used and number of executed instructions. As for memory usage the simplest way of doing this is executing DSL in separate process and measure its memory usage before execution and periodically during execution. Once memory usage goes above threshold process is terminated by master process. This method also increases security/isolation. Also note, that Murano engine is capable of simultaneous execution of many DSL-based deployments and thus hanging of one such execution doesn't block/affect other system users in any way. Personally I believe that YAQL-based DSLs similar (but probably simlet than) MuranoPL can be of a great value for many OpenStack projects that have DSLs of different kind. Murano for App Catatalog, Mistral for workflows, Heat for HOT templates, Ceilometer for alarms counter aggregation rules, Nova for customizable resource scheduling and probably many more. If you like the idea we would appreciate your contributions and help in language promotion so that together we could solve all the issues you rightly pointed out On Sat, Mar 8, 2014 at 9:05 PM, Joshua Harlow harlo...@yahoo-inc.comwrote: Thanks for the detailed explanation! I have done similar stuff to what google has done for python a long time ago for php (when rasmus, the php creator worked at y!) so I know it's not simple. The part of me that likes doing languages thinks all this muranoPL stuff is neat. But at the same time it worries me, since providing users with a avenue to run code (or a DSL that has similar concepts) scares me :P Especially if it's running in python. I noted u don't provide access to lower lever python functions (besides the green thread one) so that's great. It will just be a tight rope walk to keep that DSL free of issues when it enables all those features. Are there any pointers to the execution control that it is doing? Is there a external process time limiting the user code that is running (or is it ulimits?)? With raw loop access as it appears the DSL provides it's not exactly hard to lock up a whole process (which will lock up any other greenthreads to)... How is it that u can calculate ahead of time how much memory an object will use (python doesn't exactly provide this afaik) to impose memory limits. Are deployers of murano expected to isolate there murano deployments to avoid security problems (and resource contention)? As for lua and others (I know the javascript built into java also, due to some of this being used for something similar at y!), those languages have been designed to run in very minimal modes (no access to sockets or files by default...) and there run times have been built from the ground up to have the features needed to control execution time, memory and resource access so it might not really be as hard there as u would think (although it's still not easy of course). Python afaik was never really built to have these features, to run in very minimal modes without access to things like sockets or files... (although google apparently hacked enough of them in for usage in app-engine). Anyways maybe we should/can continue this on another thread :-) Sent from my really tiny device... On Mar 8, 2014, at 2:35 AM, Stan Lagun sla...@mirantis.com wrote: This may be not proper thread to discuss MuranoPL but since I suggested Mistral to borrow some parts of it let me answer it here. Q: Why not use some existing language? A: 1. There are not so many languages that can be embedded and securely used to run code in shared environments. And it cannot be Python itself (unless you're Google
Re: [openstack-dev] MuranoPL questions?
First of all MuranoPL is not a host but hosted language. It never intended to replace Python and if Python can do the job it is probably better than MuranoPL for that job. The problem with Python is that you cannot have Python code as a part of your DSL if you need to evaluate that DSL on server-side. Using Python's eval() is not secure. And you don't have enough control on what evaled code is allowed to do. MuranoPL on the contrary are fully sandboxed. You have absolute control over what functions/methods/APIs are exposed to DSL and DSL code can do nothing except for what it allowed to do. Besides you typically do want your DSL to be domain-specific so general-purpose language like Python can be suboptimal. I don't say MuranoPL is good for all projects. It has many Murano-specific things after all. In most cases you don't need all those OOP features that MuranoPL has. But the code organization, how it uses YAML, block structures and especially YAQL expressions can be of a great value to many projects For examples of MuranoPL classes you can browse https://github.com/istalker2/MuranoDsl/tree/master/meta folder. This is my private repository that I was using to develop PoC for MuranoPL engine. We are on the way to create production-quality implementation with unit-tests etc. in regular Murano repositories on stackforge On Sun, Mar 9, 2014 at 7:33 AM, Joshua Harlow harlo...@yahoo-inc.comwrote: To continue from other thread Personally I believe that YAQL-based DSLs similar (but probably simlet than) MuranoPL can be of a great value for many OpenStack projects that have DSLs of different kind. Murano for App Catatalog, Mistral for workflows, Heat for HOT templates, Ceilometer for alarms counter aggregation rules, Nova for customizable resource scheduling and probably many more. Isn't python a better host language for said DSLs than muranoPL? I am still pretty weary of a DSL that starts to grow so many features to encompass other DSLs since then it's not really a DSL but a non-DSL, and we already have one that everyone is familiar with (python). Are there any good examples if muranoPL that I can look at that take advantage of muranoPL classes, functions, namespaces which can be compared to there python equivalents. Has such an analysis/evaluation been done? Sent from my really tiny device... ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] MuranoPL questions?
Forgot to say that the discussion was started in [openstack-dev] [Mistral] Crack at a Real life workflow thread On Sun, Mar 9, 2014 at 10:39 AM, Stan Lagun sla...@mirantis.com wrote: First of all MuranoPL is not a host but hosted language. It never intended to replace Python and if Python can do the job it is probably better than MuranoPL for that job. The problem with Python is that you cannot have Python code as a part of your DSL if you need to evaluate that DSL on server-side. Using Python's eval() is not secure. And you don't have enough control on what evaled code is allowed to do. MuranoPL on the contrary are fully sandboxed. You have absolute control over what functions/methods/APIs are exposed to DSL and DSL code can do nothing except for what it allowed to do. Besides you typically do want your DSL to be domain-specific so general-purpose language like Python can be suboptimal. I don't say MuranoPL is good for all projects. It has many Murano-specific things after all. In most cases you don't need all those OOP features that MuranoPL has. But the code organization, how it uses YAML, block structures and especially YAQL expressions can be of a great value to many projects For examples of MuranoPL classes you can browse https://github.com/istalker2/MuranoDsl/tree/master/meta folder. This is my private repository that I was using to develop PoC for MuranoPL engine. We are on the way to create production-quality implementation with unit-tests etc. in regular Murano repositories on stackforge On Sun, Mar 9, 2014 at 7:33 AM, Joshua Harlow harlo...@yahoo-inc.comwrote: To continue from other thread Personally I believe that YAQL-based DSLs similar (but probably simlet than) MuranoPL can be of a great value for many OpenStack projects that have DSLs of different kind. Murano for App Catatalog, Mistral for workflows, Heat for HOT templates, Ceilometer for alarms counter aggregation rules, Nova for customizable resource scheduling and probably many more. Isn't python a better host language for said DSLs than muranoPL? I am still pretty weary of a DSL that starts to grow so many features to encompass other DSLs since then it's not really a DSL but a non-DSL, and we already have one that everyone is familiar with (python). Are there any good examples if muranoPL that I can look at that take advantage of muranoPL classes, functions, namespaces which can be compared to there python equivalents. Has such an analysis/evaluation been done? Sent from my really tiny device... ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Incubation Request: Murano
is XML based and it is quite painful to write a workflows without any additional instrument like IDE. It so happens that the OASIS's TOSCA technical committee are working as we speak on a TOSCA Simple Profile that will hopefully make things easier to use and includes a YAML representation (the latter is great IMHO, but the key to being able to do it is the former). Work is still at a relatively early stage and in my experience they are very much open to input from implementers. I would strongly encourage you to get involved in this effort (by joining the TOSCA TC), and also to architect Murano in such a way that it can accept input in multiple formats (this is something we are making good progress toward in Heat). Ideally the DSL format for Murano+Heat should be a trivial translation away from the relevant parts of the YAML representation of TOSCA Simple Profile. cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] Object-oriented approach for defining Murano Applications
for PrimaryController and SecondaryControllers and a Deploy workflow which call appropriate workflows on the controllers. The simplified class diagram may look like this: So, this approach allows to decompose the AD deployment workflow into simple isolated parts, explicitly manage the state and create reusable entities (of course classes like Instance, WindowsInstance, DomainMember may be used by other Murano Applications). For me this looks much, much better than the current implicit state machine which we run based on XML rules. What do you think about this approach, folks? Do you think it will be easily understood by application developers? Will it be easy to write workflows this way? Do you see any drawbacks here? Waiting for your feedback. -- Regards, Alexander Tivelkov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] Heat resource isolation withing single stack
I would like the Murano project to evaluate the upcoming heat software-config resources to represent app configuration. Yes, we are closely watching progress being made in Heat in that area and evaluating how all new great features can be used in Murano Inferring from the above, it sounds like Murano is transforming an app definition into a single flat heat template. It is just not that simple. Murano by itself neither generates any templates nor deploys applications. Murano is just App Catalog. Application definitions that Catalog consists of are more then just a declarations. They also contains imperative executable parts written in Murano DSL language. Murano itself can be viewed as a runtime for application manifests. Murano engine provides carefully designed APIs for DSL code to access Heat and other OpenStack services. Those APIs are more restrictive and safe than native OpenStack APIs (provided by various python-clients) and DSL cannot access OpenStack services by any means other than those APIs. Heat API is one of them. With this API DSL can access Heat stack and make some changes in it. It is up to Murano users (software publishers) who fill AppCatalog to decide how and if to use Heat. They are free to use Heat just for infrastructure-level management and do software config by some external tools or use new software-config in Heat or even directly talk to Nova, Trove etc. But no matter how application decides to implement it, Murano must guarantee that DSL code of one app cannot damage or compromise OpenStack resources that belongs to another app. So even if new Heat feature makes life easier we still need to guarantee safe behavior of DSL when using old Heat template syntax or older Heat version that doesn't have that feature. On Sun, Feb 23, 2014 at 11:50 PM, Steve Baker sba...@redhat.com wrote: On 22/02/14 03:37, Stan Lagun wrote: Hi Everyone, While looking through Heat templates generation code in Murano I've realized it has a major design flaw: there is no isolation between Heat resources generated by different apps. Every app manifest can access and modify its environment stack in any way. For example it can delete instances and other resources belonging to other applications. This may be not so bad for Murano 0.4 but it becomes critical for AppCatalog (0.5) as there is no trust relations between applications and it may be unacceptable that untrusted application can gain complete write access over the whole stack. There is also a problem of name collisions - resources generated by different applications may have the same names. This is especially probable between resources generated by different instances of the same app. This also affects Parameters/Output of Heat templates as each application instance must generate unique names for them (and do not forget them later as they are needed to read output results). I think we need at least to know how we going to solve it before 0.5 Here is possible directions i can think of: 1. Use nested Heat stacks. I'm not sure it solves naming collisions and that nested stacks can have their own Output 2. Control all stack template modifications and track which resource was created by which app. Give applications read-only access to resources they don't own 3. Auto-generate resource names. Auto-add prefixes/suffixes to resource/output etc names indicating owning app instance ID and remove them upon read access from workflow so that generated names would be invisible to workflow. That would also mean all VMs would have generated names Hope to see better ideas and suggestions in this thread I would like the Murano project to evaluate the upcoming heat software-config resources to represent app configuration. Some patches are still in review [1][2] and I expect more changes before feature freeze, but now would be a good time for Murano to get involved to validate that software-config meets your needs. The current implementation of the SoftwareDeployment resource actually has the same issue mentioned above; any authenticated user can currently create a deployment which results in software being deployed to any server in that tenant/project. There may be times where this is useful, but there will soon be a default policy enforced which ensures that the deployment resource and the server resource belong to the same stack. Inferring from the above, it sounds like Murano is transforming an app definition into a single flat heat template. Using a combination of resource providers and software config should give you a significantly cleaner model, and you may even find that some of the problems you're solving in Murano are now solved in the Heat layer. [1] https://review.openstack.org/#/q/status:open+topic:bp/hot-software-config-rest,n,z [2] https://review.openstack.org/#/q/status:open+topic:bp/hot-software-config,n,z
[openstack-dev] [Murano] Heat resource isolation withing single stack
Hi Everyone, While looking through Heat templates generation code in Murano I've realized it has a major design flaw: there is no isolation between Heat resources generated by different apps. Every app manifest can access and modify its environment stack in any way. For example it can delete instances and other resources belonging to other applications. This may be not so bad for Murano 0.4 but it becomes critical for AppCatalog (0.5) as there is no trust relations between applications and it may be unacceptable that untrusted application can gain complete write access over the whole stack. There is also a problem of name collisions - resources generated by different applications may have the same names. This is especially probable between resources generated by different instances of the same app. This also affects Parameters/Output of Heat templates as each application instance must generate unique names for them (and do not forget them later as they are needed to read output results). I think we need at least to know how we going to solve it before 0.5 Here is possible directions i can think of: 1. Use nested Heat stacks. I'm not sure it solves naming collisions and that nested stacks can have their own Output 2. Control all stack template modifications and track which resource was created by which app. Give applications read-only access to resources they don't own 3. Auto-generate resource names. Auto-add prefixes/suffixes to resource/output etc names indicating owning app instance ID and remove them upon read access from workflow so that generated names would be invisible to workflow. That would also mean all VMs would have generated names Hope to see better ideas and suggestions in this thread -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] Heat resource isolation withing single stack
Steve, thank you for very valuable suggestions. Your block post is really
great - I've read about environments in Heat documentation but didn't
really understood them until now.
Usage of nested stacks may or may not solve my problem depending on what is
possible to do within those stacks.
Let me explain with simple example.
As you probably know Murano uses Heat for all infrastructure-related
operations. This means if some application from Catalog needs VM instance
or any other type of OpenStack resource it creates it by inserting a
snippet into user's Heat stack template and executes UPDATE STACK command.
Now suppose there is WordPress application published in App Catalog.
WordPress app manifest says that it requires installation of MySql. There
is also another application in AppCatalog called GaleraMySql that is known
to be compatible with MySql. In Murano Dashboard user creates new
environment (this corresponds to Heat stack and is not related to what is
called environment in Heat)
and puts WordPress and GaleraMySql on it. Then he connects them so that
GaleraMySql instance would be used in WordPress for MySql requirement.
WordPress and GaleraMySql were developed by different vendors that are not
aware of each others presence. But because of unfortunate combination of
circumstances both vendors chose to merge exactly the same snippet into
user's stack:
Resources: {
myHost: {
Type: AWS::EC2::Instance,
Properties: {
InstanceType: large,
ImageId: someImage
}
}
}
Then instead of 2 different VMs there would be only one. Things would be
even worse if there was already resource myHost in user's stack.
It is more than a name-collision problem as incorrectly written application
manifest can cause any imaginable harm to the stack.
The obvious solution would be to give each app dedicated nested stack and
restrict it to that nested stack only. This would be a best solution. All I
need is to have the same level of control on nested stack I have on outer
stack - get stack template, modify and update them, access output
attributes. Is it possible to retrieve nested stack template, modify it and
populate it back to Heat?
Another option would be create separate top-level stacks for each app. But
in Murano applications themselves composed of smaller parts and in practice
this would lead to creation of dozen stacks with most of them containing
single resource. And then we would have to implement transaction update
between several stacks, coordinated deletion etc. This would also be bad
from a user's point of view at he doesn't expect to find long list of
stacks he has no idea where they came from.
My other options were on how nested stacks can be emulated on top of single
stack by controlling which app created which resource and dynamically
adjust resource names back and forth (myHost in example above) to some
unique values in a way that is opaque to application
On Fri, Feb 21, 2014 at 8:20 PM, Steven Hardy sha...@redhat.com wrote:
On Fri, Feb 21, 2014 at 06:37:27PM +0400, Stan Lagun wrote:
Hi Everyone,
While looking through Heat templates generation code in Murano I've
realized it has a major design flaw: there is no isolation between Heat
resources generated by different apps.
Can you define the requirement for isolation in more detail? Are you
referring simply to namespace isolation, or do you need auth level
isolation, e.g something enforced via keystone?
Every app manifest can access and modify its environment stack in any
way.
For example it can delete instances and other resources belonging to
other
applications. This may be not so bad for Murano 0.4 but it becomes
critical
for AppCatalog (0.5) as there is no trust relations between applications
and it may be unacceptable that untrusted application can gain complete
write access over the whole stack.
All requests to Heat are scoped by tenant/project, so unless you enforce
resource-level access policy (which we sort-of started looking at with
OS::Heat::AccessPolicy), this is expected behavior.
There is also a problem of name collisions - resources generated by
different applications may have the same names. This is especially
probable
between resources generated by different instances of the same app. This
also affects Parameters/Output of Heat templates as each application
instance must generate unique names for them (and do not forget them
later
as they are needed to read output results).
A heirarchy of nested stacks, with each application defined as a separate
stack seems the obvious solution here.
I think we need at least to know how we going to solve it before 0.5
Here is possible directions i can think of:
1. Use nested Heat stacks. I'm not sure it solves naming collisions and
that nested stacks can have their own Output
I think it does, and yes all stacks can have their own outputs
Re: [openstack-dev] [Murano] Need a new DSL for Murano
to write workflows for application deployment and maintenance. Current version of workflow definition markup really have some design drawbacks which limit its potential adoption. They are caused by the fact that it was never intended for use for Application Catalog use-cases. Just curious, is there any reason you're not collaborating on Mistral for this rather than both having a workflow engine? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Murano] Need a new DSL for Murano
it gets u none of the benefits of syntax checking, parsing, validation (highlighting...) and all the pain of yaml. Something doesn't seem right about the approach of creating languages inside the yaml format (in a way it becomes like xsl, yet xsl at least has a spec and is well defined). My 2 cents Sent from my really tiny device... On Feb 14, 2014, at 7:22 PM, Alexander Tivelkov ativel...@mirantis.com wrote: Hi folks, Murano matures, and we are getting more and more feedback from our early adopters. The overall reception is very positive, but at the same time there are some complaints as well. By now the most significant complaint is is hard to write workflows for application deployment and maintenance. Current version of workflow definition markup really have some design drawbacks which limit its potential adoption. They are caused by the fact that it was never intended for use for Application Catalog use-cases. I'll briefly touch these drawbacks first: 1. Murano's workflow engine is actually a state machine, however the workflow markup does not explicitly define the states and transitions. 2. There is no data isolation within any environment, which causes both potential security vulnerabilities and unpredictable workflow behaviours. 3. There is no easy way to reuse the workflows and their related procedures between several applications. 4. The markup uses JSONPath, which relies on Python's 'eval' function. This is insecure and has to be avoided. 5. 5. The workflow markup is XML-based, which is not a common practice in the OpenStack community. So, it turns out that we have to design and implement a new workflow definition notation, which will not have any of the issues mentioned above. At the same time, it should still allow to fully specify the configuration of any third-party Application, its dependencies with other Applications and define specific actions which are required for Application deployment, configuration and life cycle management. This new notation should allow to do the following: - List all the required configuration parameters and dependencies for a given application - Validate user input and match it to the defined parameters - Define specific deployment actions and their execution order - Define behaviors to handle the events of changes in application's environment Also, it should satisfy the following requirements: - Minimize the amount of configuration for common application parts, i.e. reuse existing configuration parts and add only difference specific to the application. - Allow to use different deployment tools with using the same markup constructs. i.e. provide a high-level abstraction on the underlying tools (heat, shell, chef, puppet etc) - For security reasons it should NOT allow to execute arbitrary operations - i.e. should allow to run only predefined set of meaningful configuration actions. So, I would suggest to introduce a simple and domain specific notation which would satisfy these needs: - Application dependencies and configuration properties are defined declaratively, in a way similar to how it is done in Heat templates. - Each property has special constraints and rules, allowing to validate the input and applications relationship within the environment. - The workflows are defined in imperative way: as a sequence of actions or method calls. This may include assigning data variables or calling the workflows of other applications. - All of these may be packaged in a YAML format. The example may look something like this [1] The final version may become a bit more complicated, but as the starting point this should look fine. I suggest to cover this in more details on our next IRC meeting on Tuesday. Any feedback or suggestions are appreciated. [1] https://etherpad.openstack.org/p/murano-new-dsl-example -- Regards, Alexander Tivelkov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [openstack-dev] Designing and implementing Murano (App Catalog for OpenStack)
Hello, As far as I see it now here is what will be required: 1. Ability to integrate support for Murano guest VM agent (probably as a component type plugin). 2. Ability to control configuration actions order by explicitly specifying pre-requirements for each component 3. All configuration actions that has all pre-requirements met need to be executed simultaneously (green threads are ok). So that we can do mass-configuration things like install XX on 100 machines (using 100 components in generated template) 4. Components need to have metadata and agent plugin implementation need to have access to it so that the values could be passed to Murano Agent. Standard HOT value referencing syntax must be supported here as well. 5. Configuration actions performed by component may produce output values that need to be accessible to other components later in dependency chain. This may be in form of explicit output attributes (as in HOT template itself) or by altering own metadata values in runtime 6. Ability to stop orchestration and/or rollback stack on configuration failures 7. Ability to modify component graph for deployed stack and thus execute some additional configuration actions on VMs created earlier. Does this fit your vision of Heat orchestration? On Fri, Nov 15, 2013 at 1:11 AM, Steve Baker sba...@redhat.com wrote: On 11/15/2013 08:30 AM, Alexander Tivelkov wrote: Hey Stackers, During HK summit we’ve been discussing an approach to implement app catalog for Openstack. We’ve seen certain interest in the topic and heard from few people that they are working on similar concepts outside of community. Apparently Openstack is now mature enough to have a need for an application catalog which will fulfil several needs of cloud admins and users on platform level: - an integration point for applications and services running on top of OS - the level of management of lifecycle of applications, distribution channels and billing - self-service provisioning for applications Few weeks ago we proposed to extend the mission of Murano to be an Application Catalog for Openstack, simultaneously we decided to contribute several parts of the project to other OpenStack projects, mainly Heat and Mistral. Our intention is to make the mission and scope of the project as lean as possible, at the same time leveraging as much of the existing Openstack services as possible. As many others we’re excited about Solum project and looking forward to integrate with Solum and contribute there. Now the most exciting part - we are looking for contributors! Use cases, architecture, blueprints, code - anything would be helpful. Right now is the best time to start, as we are just started figuring out definition of Application Catalog, getting the requirements, considering use cases and design concepts. The project from day one is run on OpenStack infrastructure, following all standard OpenStack processes and practices and we’re really making an effort to make it very open. We believe that this is an initiative which can bring a lot of value to the community, let’s try to obtain synergy working on this all together. The earlier you express your interest explicitly, the more impact you’re going to have on the roadmap. The high-level overview of the proposal can be found at our wiki: https://wiki.openstack.org/wiki/Murano/ApplicationCatalog This is a work-in-progress document, and it is going to be changed - so your feedback is very welcome right now. We’ve created several etherpads to collaborate and gather the requirements together. Feel free to add anything which seems important for you: https://etherpad.openstack.org/p/AppCatalogUI - requirements for UI https://etherpad.openstack.org/p/AppCatalogRoadmap - roadmap with features and milestones https://etherpad.openstack.org/p/AppCatalogUseCases - some use-cases, giving examples of how different users interact with the Application Catalog Feel free to add anything to these etherpads - and share any ideas you might have with all the community. Looking forward for your participation! Can you outline what might need to be added to Heat to support Murano? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] HOT Software configuration proposal
by making the system converge toward the desirable end-state through running idempotent recipes. This is our approach. The first configuration phase handles parametrization which in general brings an instance to CREATE_COMPLETE state. A second phase follows to handle contextualization at the stack level. As a matter of fact, a new contextualization should be triggered every time an instance enters or leave the CREATE_COMPLETE state which may happen any time with auto-scaling. In that phase, circular dependencies can be resolved because all contextualization data can be compiled globally. Notice that Heat doesn't provide a purpose built resource or service like Chef's data-bag for the storage and retrieval of metadata. This a gap which IMO should be addressed in the proposal. Currently, we use a kludge that is to create a fake AWS::AutoScaling::LaunchConfiguration resource to store contextualization data in the metadata section of that resource. Aside from the HOT software configuration proposal(s). There are two critical enhancements in Heat that would make software life-cycles management much easier. In fact, they are actual blockers for us. The first one would be to support asynchronous notifications when an instance is created or deleted as a result of an auto-scaling decision. As stated earlier, contextualization needs to apply in a stack every time a instance enters or leaves the CREATE_COMPLETE state. I am not referring to a Ceilometer notification but a Heat notification that can be consumed by a Heat client. The second one would be to support a new type of AWS::IAM::User (perhaps OS::IAM::User) resource whereby one could pass Keystone credentials to be able to specify Ceilometer alarms based on application's specific metrics (a.k.a KPIs). I hope this is making sense to you and can serve as a basis for further discussions and refinements. Cheers, Patrick On 10/16/13 12:48 AM, Steve Baker wrote: I've just written some proposals to address Heat's HOT software configuration needs, and I'd like to use this thread to get some feedback: https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config https://wiki.openstack.org/wiki/Heat/Blueprints/native-tools-bootstrap-config Please read the proposals and reply to the list with any comments or suggestions. We can spend some time discussing software configuration at tomorrow's Heat meeting, but I fully expect we'll still be in the discussion phase at Hong Kong. cheers ___ OpenStack-dev mailing listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Patrick Petit Cloud Computing Principal Architect, Innovative Products Bull, Architect of an Open World TM Tél : +33 (0)4 76 29 70 31 Mobile : +33 (0)6 85 22 06 39http://www.bull.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Sincerely yours Stanislav (Stan) Lagun Senior Developer Mirantis 35b/3, Vorontsovskaya St. Moscow, Russia Skype: stanlagun www.mirantis.com sla...@mirantis.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] HOT Software configuration proposal
Hello,
I've been reading through the thread and the wiki pages and I'm still
confused by the terms. Is there a clear definition of what do we understand
by component from user's and developer's point of view. If I write
component, type:MySQL what is behind that definition? I mean how does the
system know what exactly MySQL is and how to install it? What MySQL version
is it gonna be? Will it be x86 or x64? How does the system understand that
I need MySQL for Windows on Windows VM rather then Linux MySQL? What do I
as a developer need to do so that it would be possible to have type:
MyCoolComponentType?
On Tue, Oct 22, 2013 at 8:35 PM, Thomas Spatzier thomas.spatz...@de.ibm.com
wrote:
Zane Bitter zbit...@redhat.com wrote on 22.10.2013 17:23:52:
From: Zane Bitter zbit...@redhat.com
To: openstack-dev@lists.openstack.org,
Date: 22.10.2013 17:26
Subject: Re: [openstack-dev] [Heat] HOT Software configuration proposal
On 22/10/13 16:35, Thomas Spatzier wrote:
Zane Bitter zbit...@redhat.com wrote on 22.10.2013 15:24:28:
From: Zane Bitter zbit...@redhat.com
To: openstack-dev@lists.openstack.org,
Date: 22.10.2013 15:27
Subject: Re: [openstack-dev] [Heat] HOT Software configuration
proposal
On 22/10/13 09:15, Thomas Spatzier wrote:
BTW, the convention of properties being input and attributes being
output,
i.e. that subtle distinction between properties and attributes is not
really intuitive, at least not to me as non-native speaker, because I
used
to use both words as synonyms.
As a native speaker, I can confidently state that it's not intuitive
to
anyone ;)
Phew, good to read that ;-)
We unfortunately inherited these names from the Properties section and
the Fn::GetAtt function in cfn templates. It's even worse than that,
because there's a whole category of... uh... things (DependsOn,
DeletionPolicy, c.) that don't even have a name - I always have to
resist the urge to call them 'attributes' too.
So is this something we should try to get straight in HOT while we
still
have the flexibility?
Y-yes. Provided that we can do it without making things *more*
confusing, +1. That's hard though, because there are a number of places
we have to refer to them, all with different audiences:
- HOT users
- cfn users
- Existing developers
- New developers
- Plugin developers
and using different names for the same thing can cause problems. My test
for this is: if you were helping a user on IRC debug an issue, is there
a high chance you would spend 15 minutes talking past each other because
they misunderstand the terminology?
Hm, good point. Seems like it would really cause more confusion than it
helps. So back away from the general idea of renaming things that exist
both in cfn and HOT.
What we should try of course is to give new concepts that will only exist
in HOT intuitive names.
Regarding properties/attributes for example, to me I would call both
just
properties of a resource or component, and then I can write them or
read
them like:
components:
my_component:
type: ...
properties:
my_prop: { get_property: [ other_component,
other_component_prop ] }
other_component:
# ...
I.e. you write property 'my_prop' of 'my_component' in its properties
section, and you read property 'other_component_prop' of
'other_component'
using the get_property function.
... we can also call them attributes, but use one name, not two
different
names for the same thing.
IMO inputs (Properties) and outputs (Fn::GetAtt) are different things
(and they exist in different namespaces), so -1 for giving them the same
name.
In an ideal world I'd like HOT to use something like get_output_data (or
maybe just get_data), but OTOH we have e.g. FnGetAtt() and
attributes_schema baked in to the plugin API that we can't really
change, so it seems likely to lead to developers and users adopting
different terminology, or making things very difficult for new
developers, or both :(
cheers,
Zane.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Sincerely yours
Stanislav (Stan) Lagun
Senior Developer
Mirantis
35b/3, Vorontsovskaya St.
Moscow, Russia
Skype: stanlagun
www.mirantis.com
sla...@mirantis.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
