Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
On Thu, Nov 13, 2014 at 4:00 AM, Clint Byrum wrote: > Excerpts from Zane Bitter's message of 2014-11-12 08:42:44 -0800: > > On 12/11/14 10:10, Clint Byrum wrote: > > > Excerpts from Zane Bitter's message of 2014-11-11 13:06:17 -0800: > > >> On 11/11/14 13:34, Ryan Brown wrote: > > >>> I am strongly against allowing arbitrary Javascript functions for > > >>> complexity reasons. It's already difficult enough to get meaningful > > >>> errors when you up your YAML syntax. > > >> > > >> Agreed, and FWIW literally everyone that Clint has pitched the JS idea > > >> to thought it was crazy ;) > > >> > > > > > > So far nobody has stepped up to defend me, > > > > I'll defend you, but I can't defend the idea :) > > > > > so I'll accept that maybe > > > people do think it is crazy. What I'm really confused by is why we have > > > a new weird ugly language like YAQL (sorry, it, like JQ, is hideous), > > > > Agreed, and appealing to its similarity with Perl or PHP (or BASIC!) is > > probably not the way to win over Python developers :D > > > > > and that would somehow be less crazy than a well known mature language > > > that has always been meant for embedding such as javascript. > > > > JS is a Turing-complete language, it's an entirely different kettle of > > fish to a domain-specific language that is inherently safe to interpret > > from user input. Sure, we can try to lock it down. It's a very tricky > > job to get right. (Plus it requires a new external dependency of unknown > > quality... honestly if you're going to embed a Turing-complete language, > > Python is a much more obvious choice than JS.) > > > > There's a key difference though. Python was never designed to be run > from untrusted sources. Javascript was _from the beginning_. There are > at least two independent javascript implementations which both have been > designed from the ground up to run code from websites in the local > interpreter. From the standpoint of Heat, it would be even easier to do > this. > > Perhaps I can carve out some of that negative-1000-days of free time I > have and I can make it a resource plugin, with the properties being code > and references to other resources, and the attributes being the return. > > > > Anyway, I'd prefer YAQL over trying to get the intrinsic functions in > > > HOT just right. Users will want to do things we don't expect. I say, > let > > > them, or large sections of the users will simply move on to something > > > else. > > > > The other side of that argument is that users are doing one of two > > things with data they have obtained from resources in the template: > > > > 1) Passing data to software deployments > > 2) Passing data to other resources > > > > In case (1) they can easily transform the data into whatever format they > > want using their own scripts, running on their own server. > > > > In case (2), if it's not easy for them to just do what they want without > > having to perform this kind of manipulation, we have failed to design > > good resources. And if we give people the tools to just paper over the > > problem, we'll never hear about it so we can correct it at the source, > > just launch a thousand hard-to-maintain hacks into the world. > > > case (3) is trying to write a half useful template resource. With what we have this is very difficult. I think for non-trivial templates people very quickly run into the limitations of HOT. > > I for one would rather serve the users than ourselves, and preventing > them from papering over the problems so they have to whine at us is a > self-serving agenda. > > As a primary whiner about Heat for a long time, I respect a lot that > this development team _bends over backwards_ to respond to user > requests. It's amazing that way. > > However, I think to grow beyond open source savvy, deeply integrated > users like me, one has to let the users solve their own problems. They'll > know that their javascript or YAQL is debt sometimes, and they can > come to Heat's development community with suggestions like "If you had > a coalesce function I wouldn't need to write it in javascript". But if > you don't give them _something_, they'll just move on. > Agree, I think we need to get this done. We can't just keep ignoring users when they are begging for the same feature, because supposedly they are doing it wrong. -Angus > > Anyway, probably looking further down the road than I need to, but > please keep an open mind for this idea, as users tend to use tools that > solve their problem _and_ get out of their way in all other cases. > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
Will be very interesting to see how this plays out. FYI, yql (a thing yahoo uses a-lot of, internally and externally) has this kind of functionality built-in. https://developer.yahoo.com/yql/guide/yql-execute-chapter.html I am unaware of any python version of a javascript engine that has rate/execution limiting and such[2], so I'm not sure how feasible this is without switching languages (to java using rhino[3] or one of the firefox monkey engines or chromes engine...). It would though be neat to expose javascript functions to heat so that people could extend heat where they felt they needed to and 'glue' together there various javascript functions into a larger orchestration template that heat manages (and/or use native heat functionality that exists). That would seem like a good mix of functionality for people to have (and enables more advanced users); although it does of course introduce a bunch of complexity (first being where does this javascript code get ran, where is it stored...). [2] https://developer.yahoo.com/yql/guide/yql-execute-intro-ratelimits.html[3] http://en.wikipedia.org/wiki/Rhino_%28JavaScript_engine%29 -Josh From: Clint Byrum To: openstack-dev Sent: Wednesday, November 12, 2014 10:00 AM Subject: Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull Excerpts from Zane Bitter's message of 2014-11-12 08:42:44 -0800: > On 12/11/14 10:10, Clint Byrum wrote: > > Excerpts from Zane Bitter's message of 2014-11-11 13:06:17 -0800: > >> On 11/11/14 13:34, Ryan Brown wrote: > >>> I am strongly against allowing arbitrary Javascript functions for > >>> complexity reasons. It's already difficult enough to get meaningful > >>> errors when you up your YAML syntax. > >> > >> Agreed, and FWIW literally everyone that Clint has pitched the JS idea > >> to thought it was crazy ;) > >> > > > > So far nobody has stepped up to defend me, > > I'll defend you, but I can't defend the idea :) > > > so I'll accept that maybe > > people do think it is crazy. What I'm really confused by is why we have > > a new weird ugly language like YAQL (sorry, it, like JQ, is hideous), > > Agreed, and appealing to its similarity with Perl or PHP (or BASIC!) is > probably not the way to win over Python developers :D > > > and that would somehow be less crazy than a well known mature language > > that has always been meant for embedding such as javascript. > > JS is a Turing-complete language, it's an entirely different kettle of > fish to a domain-specific language that is inherently safe to interpret > from user input. Sure, we can try to lock it down. It's a very tricky > job to get right. (Plus it requires a new external dependency of unknown > quality... honestly if you're going to embed a Turing-complete language, > Python is a much more obvious choice than JS.) > There's a key difference though. Python was never designed to be run from untrusted sources. Javascript was _from the beginning_. There are at least two independent javascript implementations which both have been designed from the ground up to run code from websites in the local interpreter. From the standpoint of Heat, it would be even easier to do this. Perhaps I can carve out some of that negative-1000-days of free time I have and I can make it a resource plugin, with the properties being code and references to other resources, and the attributes being the return. > > Anyway, I'd prefer YAQL over trying to get the intrinsic functions in > > HOT just right. Users will want to do things we don't expect. I say, let > > them, or large sections of the users will simply move on to something > > else. > > The other side of that argument is that users are doing one of two > things with data they have obtained from resources in the template: > > 1) Passing data to software deployments > 2) Passing data to other resources > > In case (1) they can easily transform the data into whatever format they > want using their own scripts, running on their own server. > > In case (2), if it's not easy for them to just do what they want without > having to perform this kind of manipulation, we have failed to design > good resources. And if we give people the tools to just paper over the > problem, we'll never hear about it so we can correct it at the source, > just launch a thousand hard-to-maintain hacks into the world. > I for one would rather serve the users than ourselves, and preventing them from papering over the problems so they have to whine at us is a self-serving agenda. As a primary whiner about Heat for a long time, I respect a lot that this devel
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
Excerpts from Zane Bitter's message of 2014-11-12 08:42:44 -0800: > On 12/11/14 10:10, Clint Byrum wrote: > > Excerpts from Zane Bitter's message of 2014-11-11 13:06:17 -0800: > >> On 11/11/14 13:34, Ryan Brown wrote: > >>> I am strongly against allowing arbitrary Javascript functions for > >>> complexity reasons. It's already difficult enough to get meaningful > >>> errors when you up your YAML syntax. > >> > >> Agreed, and FWIW literally everyone that Clint has pitched the JS idea > >> to thought it was crazy ;) > >> > > > > So far nobody has stepped up to defend me, > > I'll defend you, but I can't defend the idea :) > > > so I'll accept that maybe > > people do think it is crazy. What I'm really confused by is why we have > > a new weird ugly language like YAQL (sorry, it, like JQ, is hideous), > > Agreed, and appealing to its similarity with Perl or PHP (or BASIC!) is > probably not the way to win over Python developers :D > > > and that would somehow be less crazy than a well known mature language > > that has always been meant for embedding such as javascript. > > JS is a Turing-complete language, it's an entirely different kettle of > fish to a domain-specific language that is inherently safe to interpret > from user input. Sure, we can try to lock it down. It's a very tricky > job to get right. (Plus it requires a new external dependency of unknown > quality... honestly if you're going to embed a Turing-complete language, > Python is a much more obvious choice than JS.) > There's a key difference though. Python was never designed to be run from untrusted sources. Javascript was _from the beginning_. There are at least two independent javascript implementations which both have been designed from the ground up to run code from websites in the local interpreter. From the standpoint of Heat, it would be even easier to do this. Perhaps I can carve out some of that negative-1000-days of free time I have and I can make it a resource plugin, with the properties being code and references to other resources, and the attributes being the return. > > Anyway, I'd prefer YAQL over trying to get the intrinsic functions in > > HOT just right. Users will want to do things we don't expect. I say, let > > them, or large sections of the users will simply move on to something > > else. > > The other side of that argument is that users are doing one of two > things with data they have obtained from resources in the template: > > 1) Passing data to software deployments > 2) Passing data to other resources > > In case (1) they can easily transform the data into whatever format they > want using their own scripts, running on their own server. > > In case (2), if it's not easy for them to just do what they want without > having to perform this kind of manipulation, we have failed to design > good resources. And if we give people the tools to just paper over the > problem, we'll never hear about it so we can correct it at the source, > just launch a thousand hard-to-maintain hacks into the world. > I for one would rather serve the users than ourselves, and preventing them from papering over the problems so they have to whine at us is a self-serving agenda. As a primary whiner about Heat for a long time, I respect a lot that this development team _bends over backwards_ to respond to user requests. It's amazing that way. However, I think to grow beyond open source savvy, deeply integrated users like me, one has to let the users solve their own problems. They'll know that their javascript or YAQL is debt sometimes, and they can come to Heat's development community with suggestions like "If you had a coalesce function I wouldn't need to write it in javascript". But if you don't give them _something_, they'll just move on. Anyway, probably looking further down the road than I need to, but please keep an open mind for this idea, as users tend to use tools that solve their problem _and_ get out of their way in all other cases. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
On Nov 12, 2014, at 10:42 AM, Zane Bitter wrote: > On 12/11/14 10:10, Clint Byrum wrote: >> Excerpts from Zane Bitter's message of 2014-11-11 13:06:17 -0800: >>> On 11/11/14 13:34, Ryan Brown wrote: I am strongly against allowing arbitrary Javascript functions for complexity reasons. It's already difficult enough to get meaningful errors when you up your YAML syntax. >>> >>> Agreed, and FWIW literally everyone that Clint has pitched the JS idea >>> to thought it was crazy ;) >>> >> >> So far nobody has stepped up to defend me, > > I'll defend you, but I can't defend the idea :) > >> so I'll accept that maybe >> people do think it is crazy. What I'm really confused by is why we have >> a new weird ugly language like YAQL (sorry, it, like JQ, is hideous), > > Agreed, and appealing to its similarity with Perl or PHP (or BASIC!) is > probably not the way to win over Python developers :D > >> and that would somehow be less crazy than a well known mature language >> that has always been meant for embedding such as javascript. > > JS is a Turing-complete language, it's an entirely different kettle of fish > to a domain-specific language that is inherently safe to interpret from user > input. Sure, we can try to lock it down. It's a very tricky job to get right. > (Plus it requires a new external dependency of unknown quality... honestly if > you're going to embed a Turing-complete language, Python is a much more > obvious choice than JS.) > >> Anyway, I'd prefer YAQL over trying to get the intrinsic functions in >> HOT just right. Users will want to do things we don't expect. I say, let >> them, or large sections of the users will simply move on to something >> else. > > The other side of that argument is that users are doing one of two things > with data they have obtained from resources in the template: > > 1) Passing data to software deployments > 2) Passing data to other resources > > In case (1) they can easily transform the data into whatever format they want > using their own scripts, running on their own server. > > In case (2), if it's not easy for them to just do what they want without > having to perform this kind of manipulation, we have failed to design good > resources. And if we give people the tools to just paper over the problem, > we'll never hear about it so we can correct it at the source, just launch a > thousand hard-to-maintain hacks into the world. I disagree with this last bit. Having some manner of data manipulation facility as part of the template language doesn't mean the resources are badly designed or implemented, it just means that users have constraints that we don't directly address. Having something more general purpose (whatever it may be) seems like a longer term and more maintainable solution to me than eternal tail chasing where 1000 specific intrinsic functions bloom. IMO, intrinsics have a place, but supporting something like YAQL (or whatever) seems like a general solution to a lot of common problems our users come up against. As for "hard-to-maintain hacks", isn't that what drives a lot of our users to ask for these sorts of solutions in the first place? If doing something via YAQL is perceived by a user as some sub-optimal hack for their use case, I see no reason for that to be any less visible than any other pain point or usability issue we have. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
On 12/11/14 10:10, Clint Byrum wrote: Excerpts from Zane Bitter's message of 2014-11-11 13:06:17 -0800: On 11/11/14 13:34, Ryan Brown wrote: I am strongly against allowing arbitrary Javascript functions for complexity reasons. It's already difficult enough to get meaningful errors when you up your YAML syntax. Agreed, and FWIW literally everyone that Clint has pitched the JS idea to thought it was crazy ;) So far nobody has stepped up to defend me, I'll defend you, but I can't defend the idea :) so I'll accept that maybe people do think it is crazy. What I'm really confused by is why we have a new weird ugly language like YAQL (sorry, it, like JQ, is hideous), Agreed, and appealing to its similarity with Perl or PHP (or BASIC!) is probably not the way to win over Python developers :D and that would somehow be less crazy than a well known mature language that has always been meant for embedding such as javascript. JS is a Turing-complete language, it's an entirely different kettle of fish to a domain-specific language that is inherently safe to interpret from user input. Sure, we can try to lock it down. It's a very tricky job to get right. (Plus it requires a new external dependency of unknown quality... honestly if you're going to embed a Turing-complete language, Python is a much more obvious choice than JS.) Anyway, I'd prefer YAQL over trying to get the intrinsic functions in HOT just right. Users will want to do things we don't expect. I say, let them, or large sections of the users will simply move on to something else. The other side of that argument is that users are doing one of two things with data they have obtained from resources in the template: 1) Passing data to software deployments 2) Passing data to other resources In case (1) they can easily transform the data into whatever format they want using their own scripts, running on their own server. In case (2), if it's not easy for them to just do what they want without having to perform this kind of manipulation, we have failed to design good resources. And if we give people the tools to just paper over the problem, we'll never hear about it so we can correct it at the source, just launch a thousand hard-to-maintain hacks into the world. cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
My point is that quoting problem do exists probably but it exists even
without YAQL being used anywhere.
For example consider Mistral workbook containing value: { get_attr:
[my_instance, first_address] }. get_attr in Mistral may have nothing to to
with Heat's get_attr and even if it is it may be just a HOT snippet that
cannot be evaluated at the moment. For example it may be part of a HOT
template embedded into Mistral workbook and Mistral workflow is going to
create another Heat stack using that template. Now if you embed such
workbook into HOT template (and thus get HOT embedded into Mistral embedded
into HOT :) ) you need to make sure that Heat's parser is not going to try
to parse that workbook but treats it like a black box. This is regardless
of YAQL being even exist. But if you treat workbooks like
black-boxes/arbitrary JSON you should not care if it contains YAQL or not.
As for Murano this may cause some problems in cases when someone just took
YAML HOT and copy-paste it into MuranoPL class without double-checking. But
those errors can be easily fixed and such issues do exist even with current
HOT syntax without YAQL being used at all.
What I'm really confused by is why we have
> a new weird ugly language like YAQL (sorry, it, like JQ, is hideous),
>
YAQL syntax is almost identical to Python's or JS if you narrow them to
one-line expressions. The only noticeable difference is that all
variable/parameter names are prefixed with '$' as in PHP/Perl which are
both popular languages. I can hardly imagine why such negligible difference
from JS or Python (languages you do like) makes language "weird ugly"
Sincerely yours,
Stan Lagun
Principal Software Engineer @ Mirantis
On Wed, Nov 12, 2014 at 6:10 PM, Clint Byrum wrote:
> Excerpts from Zane Bitter's message of 2014-11-11 13:06:17 -0800:
> > On 11/11/14 13:34, Ryan Brown wrote:
> > > I am strongly against allowing arbitrary Javascript functions for
> > > complexity reasons. It's already difficult enough to get meaningful
> > > errors when you up your YAML syntax.
> >
> > Agreed, and FWIW literally everyone that Clint has pitched the JS idea
> > to thought it was crazy ;)
> >
>
> So far nobody has stepped up to defend me, so I'll accept that maybe
> people do think it is crazy. What I'm really confused by is why we have
> a new weird ugly language like YAQL (sorry, it, like JQ, is hideous),
> and that would somehow be less crazy than a well known mature language
> that has always been meant for embedding such as javascript.
>
> Anyway, I'd prefer YAQL over trying to get the intrinsic functions in
> HOT just right. Users will want to do things we don't expect. I say, let
> them, or large sections of the users will simply move on to something
> else.
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
Excerpts from Zane Bitter's message of 2014-11-11 13:06:17 -0800: > On 11/11/14 13:34, Ryan Brown wrote: > > I am strongly against allowing arbitrary Javascript functions for > > complexity reasons. It's already difficult enough to get meaningful > > errors when you up your YAML syntax. > > Agreed, and FWIW literally everyone that Clint has pitched the JS idea > to thought it was crazy ;) > So far nobody has stepped up to defend me, so I'll accept that maybe people do think it is crazy. What I'm really confused by is why we have a new weird ugly language like YAQL (sorry, it, like JQ, is hideous), and that would somehow be less crazy than a well known mature language that has always been meant for embedding such as javascript. Anyway, I'd prefer YAQL over trying to get the intrinsic functions in HOT just right. Users will want to do things we don't expect. I say, let them, or large sections of the users will simply move on to something else. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
On 12/11/14 08:24, Stan Lagun wrote:
On Wed, Nov 12, 2014 at 3:50 PM, Zane Bitter mailto:zbit...@redhat.com>> wrote:
It's actually potentially horrible, because you introduce potential
quoting issues when you embed mistral workbooks in Heat templates or
pass Heat templates to Murano.
Could you please give an example of template with such issue?
I don't have one to hand, but we'd want to prove that such an issue
*can't* happen before implementing this.
Also note that
a. YAQL uses only single quote marks (') for string while JSON uses
double quotes (")
b. If Mistral worbook is embedded into HOT template it should not be
parsed or interpreted anyway. Otherwise consider what would happen if
that workbook would in turn have HOT snippets embedded
That's exactly the problem I'm talking about. If you have a HOT snippet
embedded in a Mistral workbook embedded in a HOT template passed to
Murano, can you guarantee that each piece of YAQL will be interpreted
only in exactly the right place? Will the user be able to predict where
that will be? My brain hurts just thinking about it.
We can protect other formats from Heat by wrapping an intrinsic function
around the YAQL; I'm not certain that HOT would be protected from
Mistral and Murano in the same way, because this wasn't previously a
design goal for them.
cheers,
Zane.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
On 11/12/2014 07:50 AM, Zane Bitter wrote: > On 12/11/14 06:48, Angus Salkeld wrote: >> (it's nice that there would be a consistent user experience >> between these projects -mistral/heat/murano). > > It's actually potentially horrible, because you introduce potential > quoting issues when you embed mistral workbooks in Heat templates or > pass Heat templates to Murano. I think the intention here was to say there are two paths forward. 1) Add an intrinsic for first_nonnull, and every other task. 2) Add YAQL to let users build their own expressions. And that (1) involves users learning the Heat way of munging their data with intrinsics, while (2) encourages them to learn YAQL, which they can use across Heat, Mistral, and Murano. This would be a better experience since users would (in theory) spend less time looking up the domain language for task X. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
On Wed, Nov 12, 2014 at 3:50 PM, Zane Bitter wrote:
> It's actually potentially horrible, because you introduce potential
> quoting issues when you embed mistral workbooks in Heat templates or pass
> Heat templates to Murano.
Could you please give an example of template with such issue?
Also note that
a. YAQL uses only single quote marks (') for string while JSON uses double
quotes (")
b. If Mistral worbook is embedded into HOT template it should not be parsed
or interpreted anyway. Otherwise consider what would happen if that
workbook would in turn have HOT snippets embedded
Sincerely yours,
Stan Lagun
Principal Software Engineer @ Mirantis
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
On 12/11/14 06:48, Angus Salkeld wrote: (it's nice that there would be a consistent user experience between these projects -mistral/heat/murano). It's actually potentially horrible, because you introduce potential quoting issues when you embed mistral workbooks in Heat templates or pass Heat templates to Murano. cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
On 12/11/14 06:33, Alexis Lee wrote: >I would much prefer to resurrect the previous proposal for adding >conditionals and then see what is still missing than to just dive >straight in to embedding a whole other language in HOT. Do you mean this?https://blueprints.launchpad.net/heat/+spec/intrinsics This only provides string equality and boolean logic. It's insufficient to implement first_nonnull for example. I meant this: https://review.openstack.org/#/c/84468/ - ZB ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
On Wed, Nov 12, 2014 at 9:33 PM, Alexis Lee wrote: > Zane Bitter said on Tue, Nov 11, 2014 at 04:06:17PM -0500: > > FWIW literally everyone that Clint has pitched the JS > > idea to thought it was crazy ;) > > +1 > > > YAQL ... looks like line noise to me > > YAML representing function call stacks (an AST) looks pretty bad to me > :) > > The YAQL doc is not great at the moment but the language is not > difficult. It's pretty similar to XPath, jq and JSP EL, i.e. dotted > syntax with square brackets for attribute selection and slicing. That's > also pretty similar to Python btw. > > Is there another expression language you prefer? > > > From the beginning we've tried to have the absolute minimum > > number of intrinsic functions in HOT. > > This is understandable but there are things I want to do (as an operator > - like first_nonnull) which I cannot. The options as I see them: > > * Add a bunch of operators and a LOT of functions > * Add a bunch of operators, a few functions and lambda > * Add a bunch of functions and use an expression language for > arithmetic, comparison etc > * Use an expression language for all complex processing > I think it's pretty clear we need something more powerful that adding heaps of intrinsic functions, IMHO as long as we can prove it is safe we should add yaql (it's nice that there would be a consistent user experience between these projects -mistral/heat/murano). -Angus > > I would much prefer to resurrect the previous proposal for adding > > conditionals and then see what is still missing than to just dive > > straight in to embedding a whole other language in HOT. > > Do you mean this? https://blueprints.launchpad.net/heat/+spec/intrinsics > This only provides string equality and boolean logic. It's insufficient > to implement first_nonnull for example. > > I'd like to emphasise that we don't have to offer the whole YAQL stdlib > in HOT initially. We can use it just for its operator suite and a few > key functions then add others as usecases are discovered. > > +1 > > On 11/11/14 13:34, Ryan Brown wrote: > > >Vendored HOT > > the real issue is that it's under the control of the operator and not > > the template author. > > Yep, the solution has to be through the template as that's 99% of the > user interface. > > > Alexis > -- > Nova Engineer, HP Cloud. AKA lealexis, lxsli. > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
Zane Bitter said on Tue, Nov 11, 2014 at 04:06:17PM -0500: > FWIW literally everyone that Clint has pitched the JS > idea to thought it was crazy ;) +1 > YAQL ... looks like line noise to me YAML representing function call stacks (an AST) looks pretty bad to me :) The YAQL doc is not great at the moment but the language is not difficult. It's pretty similar to XPath, jq and JSP EL, i.e. dotted syntax with square brackets for attribute selection and slicing. That's also pretty similar to Python btw. Is there another expression language you prefer? > From the beginning we've tried to have the absolute minimum > number of intrinsic functions in HOT. This is understandable but there are things I want to do (as an operator - like first_nonnull) which I cannot. The options as I see them: * Add a bunch of operators and a LOT of functions * Add a bunch of operators, a few functions and lambda * Add a bunch of functions and use an expression language for arithmetic, comparison etc * Use an expression language for all complex processing > I would much prefer to resurrect the previous proposal for adding > conditionals and then see what is still missing than to just dive > straight in to embedding a whole other language in HOT. Do you mean this? https://blueprints.launchpad.net/heat/+spec/intrinsics This only provides string equality and boolean logic. It's insufficient to implement first_nonnull for example. I'd like to emphasise that we don't have to offer the whole YAQL stdlib in HOT initially. We can use it just for its operator suite and a few key functions then add others as usecases are discovered. > On 11/11/14 13:34, Ryan Brown wrote: > >Vendored HOT > the real issue is that it's under the control of the operator and not > the template author. Yep, the solution has to be through the template as that's 99% of the user interface. Alexis -- Nova Engineer, HP Cloud. AKA lealexis, lxsli. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
For those who don't know 100% of guys behind YAQL are also active OpenStack contributors. During (early) Kilo cycle we plan to release YAQL version 1.0 on stackforge. This release is going to fix some flaws in early versions, add some more flexibility and have very high UT coverage. There are at least 2 projects in OpenStack that already make heavy use of YAQL: Mistral and Murano. If Heat team decides to use YAQL as well we will be glad to help integrate it smoothly with HOT as well as make sure that YAQL 1.0 is aligned with Heat use cases. Sincerely yours, Stan Lagun Principal Software Engineer @ Mirantis On Wed, Nov 12, 2014 at 12:32 AM, Angus Salkeld wrote: > > > On Wed, Nov 12, 2014 at 1:35 AM, Alexis Lee wrote: > >> Alexis Lee said on Mon, Nov 10, 2014 at 05:34:13PM +: >> > How about we support YAQL expressions? >> https://github.com/ativelkov/yaql >> > Plus some HOFs (higher-order functions) like cond, map, filter, foldleft >> > etc? >> >> We could also use YAQL to provide the HOFs. >> >> > Here's first_nonnull: >> > >> > config: >> > Fn::Select >> > - 0 >> > filter: >> > - yaql: "$.0 != null" >> > - >> > - >> >> config: >> yaql: "$[$ != null][0]" >> - >> - >> >> This approach requires less change to Heat, at the price of learning >> more YAQL. >> > > +1 to YAQL > > > -Angus > > >> >> >> Alexis >> -- >> Nova Engineer, HP Cloud. AKA lealexis, lxsli. >> >> ___ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
On Wed, Nov 12, 2014 at 1:35 AM, Alexis Lee wrote: > Alexis Lee said on Mon, Nov 10, 2014 at 05:34:13PM +: > > How about we support YAQL expressions? https://github.com/ativelkov/yaql > > Plus some HOFs (higher-order functions) like cond, map, filter, foldleft > > etc? > > We could also use YAQL to provide the HOFs. > > > Here's first_nonnull: > > > > config: > > Fn::Select > > - 0 > > filter: > > - yaql: "$.0 != null" > > - > > - > > config: > yaql: "$[$ != null][0]" > - > - > > This approach requires less change to Heat, at the price of learning > more YAQL. > +1 to YAQL -Angus > > > Alexis > -- > Nova Engineer, HP Cloud. AKA lealexis, lxsli. > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
On 11/11/14 13:34, Ryan Brown wrote: I am strongly against allowing arbitrary Javascript functions for complexity reasons. It's already difficult enough to get meaningful errors when you up your YAML syntax. Agreed, and FWIW literally everyone that Clint has pitched the JS idea to thought it was crazy ;) I don't really like YAQL either though. It looks like line noise to me. From the beginning we've tried to have the absolute minimum number of intrinsic functions in HOT. I would much prefer to resurrect the previous proposal for adding conditionals and then see what is still missing than to just dive straight in to embedding a whole other language in HOT. AIUI the functionality many users would look to use Javascript embed-ability for would be better served either by something like yaql, or by making vendored HOT functions possible. Vendored HOT would look something like "X-Vendor::YourNamespace" and functions could be managed similarly to resource plugins (stevedore). It's a very rough idea, but I like it much better than adding Javascript. That already existed at one point. We ripped out the plugin part during Juno in favour of tying the intrinsic functions available to a template format, so that vendoring HOT with new intrinsic functions also requires (or at least suggests) changing the template format version string, to allow Heat can detect immediately when you are using an unsupported template format. That said it still basically works, but the real issue is that it's under the control of the operator and not the template author. cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
On 11/11/2014 01:12 PM, Clint Byrum wrote:
> Excerpts from Alexis Lee's message of 2014-11-10 09:34:13 -0800:
>> Zane Bitter said on Fri, Nov 07, 2014 at 12:35:09AM +0100:
>>> Crazy thought: why not just implement conditionals? We had a
>>> proto-spec for them started at one point...
>>
>> I didn't know that was on the table :)
>>
>> How about we support YAQL expressions? https://github.com/ativelkov/yaql
>> Plus some HOFs (higher-order functions) like cond, map, filter, foldleft
>> etc?
>>
>> Here's first_nonnull:
>>
>> config:
>> Fn::Select
>> - 0
>> filter:
>> - yaql: "$.0 != null"
>> -
>> -
>>
>> Making the 'yaql' function eponymous means we can easily plug other
>> expression languages in later if we choose.
Adding yaql (or similar, e.g. jsonpointer) seems like it would be a good
idea, and shouldn't add more complexity than it's worth.
> Cool. I think this aligns nicely with my suggestion at the summit that
> we also allow writing functions in javascript. This would allow people
> to write their own first_nonnull/coalesce until we have a chance to add
> it:
>
> config:
> embed:
> lang: javascript
> args: - item1
> - item2
> code: |
> for (arg in args) {
> if (arg !== null) {
> return arg
> }
> }
> return null;
>
> I'd really love to have this functionality but I doubt I'll have time to
> spec and land it. Does anybody else think this functionality would be a
> good way to allow template authors flexibility that they desire?
I am strongly against allowing arbitrary Javascript functions for
complexity reasons. It's already difficult enough to get meaningful
errors when you up your YAML syntax.
AIUI the functionality many users would look to use Javascript
embed-ability for would be better served either by something like yaql,
or by making vendored HOT functions possible.
Vendored HOT would look something like "X-Vendor::YourNamespace" and
functions could be managed similarly to resource plugins (stevedore).
It's a very rough idea, but I like it much better than adding Javascript.
--
Ryan Brown / Software Engineer, Openstack / Red Hat, Inc.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
Excerpts from Alexis Lee's message of 2014-11-10 09:34:13 -0800:
> Zane Bitter said on Fri, Nov 07, 2014 at 12:35:09AM +0100:
> > Crazy thought: why not just implement conditionals? We had a
> > proto-spec for them started at one point...
>
> I didn't know that was on the table :)
>
> How about we support YAQL expressions? https://github.com/ativelkov/yaql
> Plus some HOFs (higher-order functions) like cond, map, filter, foldleft
> etc?
>
> Here's first_nonnull:
>
> config:
> Fn::Select
> - 0
> filter:
> - yaql: "$.0 != null"
> -
> -
>
> Making the 'yaql' function eponymous means we can easily plug other
> expression languages in later if we choose.
>
Cool. I think this aligns nicely with my suggestion at the summit that
we also allow writing functions in javascript. This would allow people
to write their own first_nonnull/coalesce until we have a chance to add
it:
config:
embed:
lang: javascript
args: - item1
- item2
code: |
for (arg in args) {
if (arg !== null) {
return arg
}
}
return null;
I'd really love to have this functionality but I doubt I'll have time to
spec and land it. Does anybody else think this functionality would be a
good way to allow template authors flexibility that they desire?
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
On 10/11/14 12:34, Alexis Lee wrote: Zane Bitter said on Fri, Nov 07, 2014 at 12:35:09AM +0100: Crazy thought: why not just implement conditionals? We had a proto-spec for them started at one point... I didn't know that was on the table :) How about we support YAQL expressions? https://github.com/ativelkov/yaql Plus some HOFs (higher-order functions) like cond, map, filter, foldleft etc? Here's first_nonnull: config: Fn::Select - 0 filter: - yaql: "$.0 != null" - - Making the 'yaql' function eponymous means we can easily plug other expression languages in later if we choose. We can restrict the functions available to YAQL expressions to avoid a halting problem. Each HOF will know how to populate the context for subordinate YAQL expressions. When used outside a HOF, only add a default context of parameters and input_values. Is this totally crazy or worth spec-ifying? The first one. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
I like this approach and seems to have greater utility beyond the original proposal. I'm not fussed on YAQL or straight up JSONPath, but something along those lines seems to make sense. On Nov 11, 2014, at 9:35 AM, Alexis Lee wrote: > Alexis Lee said on Mon, Nov 10, 2014 at 05:34:13PM +: >> How about we support YAQL expressions? https://github.com/ativelkov/yaql >> Plus some HOFs (higher-order functions) like cond, map, filter, foldleft >> etc? > > We could also use YAQL to provide the HOFs. > >> Here's first_nonnull: >> >> config: >>Fn::Select >> - 0 >> filter: >>- yaql: "$.0 != null" >>- >>- > > config: >yaql: "$[$ != null][0]" >- >- > > This approach requires less change to Heat, at the price of learning > more YAQL. > > > Alexis > -- > Nova Engineer, HP Cloud. AKA lealexis, lxsli. > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
Alexis Lee said on Mon, Nov 10, 2014 at 05:34:13PM +: > How about we support YAQL expressions? https://github.com/ativelkov/yaql > Plus some HOFs (higher-order functions) like cond, map, filter, foldleft > etc? We could also use YAQL to provide the HOFs. > Here's first_nonnull: > > config: > Fn::Select > - 0 > filter: > - yaql: "$.0 != null" > - > - config: yaql: "$[$ != null][0]" - - This approach requires less change to Heat, at the price of learning more YAQL. Alexis -- Nova Engineer, HP Cloud. AKA lealexis, lxsli. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull
Zane Bitter said on Fri, Nov 07, 2014 at 12:35:09AM +0100: > Crazy thought: why not just implement conditionals? We had a > proto-spec for them started at one point... I didn't know that was on the table :) How about we support YAQL expressions? https://github.com/ativelkov/yaql Plus some HOFs (higher-order functions) like cond, map, filter, foldleft etc? Here's first_nonnull: config: Fn::Select - 0 filter: - yaql: "$.0 != null" - - Making the 'yaql' function eponymous means we can easily plug other expression languages in later if we choose. We can restrict the functions available to YAQL expressions to avoid a halting problem. Each HOF will know how to populate the context for subordinate YAQL expressions. When used outside a HOF, only add a default context of parameters and input_values. Is this totally crazy or worth spec-ifying? Alexis -- Nova Engineer, HP Cloud. AKA lealexis, lxsli. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
