Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

2014-11-13 Thread Joshua Harlow
Will be very interesting to see how this plays out.
FYI, yql (a thing yahoo uses a-lot of, internally and externally) has this kind 
of functionality built-in.
https://developer.yahoo.com/yql/guide/yql-execute-chapter.html
I am unaware of any python version of a javascript engine that has 
rate/execution limiting and such[2], so I'm not sure how feasible this is 
without switching languages (to java using rhino[3] or one of the firefox 
monkey engines or chromes engine...).

It would though be neat to expose javascript functions to heat so that people 
could extend heat where they felt they needed to and 'glue' together there 
various javascript functions into a larger orchestration template that heat 
manages (and/or use native heat functionality that exists). That would seem 
like a good mix of functionality for people to have (and enables more advanced 
users); although it does of course introduce a bunch of complexity (first being 
where does this javascript code get ran, where is it stored...).

[2] https://developer.yahoo.com/yql/guide/yql-execute-intro-ratelimits.html[3] 
http://en.wikipedia.org/wiki/Rhino_%28JavaScript_engine%29

-Josh

 From: Clint Byrum cl...@fewbar.com
 To: openstack-dev openstack-dev@lists.openstack.org 
 Sent: Wednesday, November 12, 2014 10:00 AM
 Subject: Re: [openstack-dev] [Heat] Conditionals, was: New function: 
first_nonnull
   
Excerpts from Zane Bitter's message of 2014-11-12 08:42:44 -0800:
 On 12/11/14 10:10, Clint Byrum wrote:
  Excerpts from Zane Bitter's message of 2014-11-11 13:06:17 -0800:
  On 11/11/14 13:34, Ryan Brown wrote:
  I am strongly against allowing arbitrary Javascript functions for
  complexity reasons. It's already difficult enough to get meaningful
  errors when you  up your YAML syntax.
 
  Agreed, and FWIW literally everyone that Clint has pitched the JS idea
  to thought it was crazy ;)
 
 
  So far nobody has stepped up to defend me,
 
 I'll defend you, but I can't defend the idea :)
 
  so I'll accept that maybe
  people do think it is crazy. What I'm really confused by is why we have
  a new weird ugly language like YAQL (sorry, it, like JQ, is hideous),
 
 Agreed, and appealing to its similarity with Perl or PHP (or BASIC!) is 
 probably not the way to win over Python developers :D
 
  and that would somehow be less crazy than a well known mature language
  that has always been meant for embedding such as javascript.
 
 JS is a Turing-complete language, it's an entirely different kettle of 
 fish to a domain-specific language that is inherently safe to interpret 
 from user input. Sure, we can try to lock it down. It's a very tricky 
 job to get right. (Plus it requires a new external dependency of unknown 
 quality... honestly if you're going to embed a Turing-complete language, 
 Python is a much more obvious choice than JS.)
 

There's a key difference though. Python was never designed to be run
from untrusted sources. Javascript was _from the beginning_. There are
at least two independent javascript implementations which both have been
designed from the ground up to run code from websites in the local
interpreter. From the standpoint of Heat, it would be even easier to do
this.

Perhaps I can carve out some of that negative-1000-days of free time I
have and I can make it a resource plugin, with the properties being code
and references to other resources, and the attributes being the return.

  Anyway, I'd prefer YAQL over trying to get the intrinsic functions in
  HOT just right. Users will want to do things we don't expect. I say, let
  them, or large sections of the users will simply move on to something
  else.
 
 The other side of that argument is that users are doing one of two 
 things with data they have obtained from resources in the template:
 
 1) Passing data to software deployments
 2) Passing data to other resources
 
 In case (1) they can easily transform the data into whatever format they 
 want using their own scripts, running on their own server.
 
 In case (2), if it's not easy for them to just do what they want without 
 having to perform this kind of manipulation, we have failed to design 
 good resources. And if we give people the tools to just paper over the 
 problem, we'll never hear about it so we can correct it at the source, 
 just launch a thousand hard-to-maintain hacks into the world.
 

I for one would rather serve the users than ourselves, and preventing
them from papering over the problems so they have to whine at us is a
self-serving agenda.

As a primary whiner about Heat for a long time, I respect a lot that
this development team _bends over backwards_ to respond to user
requests. It's amazing that way.

However, I think to grow beyond open source savvy, deeply integrated
users like me, one has to let the users solve their own problems. They'll
know that their javascript or YAQL is debt sometimes, and they can
come to Heat's development community with suggestions like If you had
a coalesce

Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

2014-11-13 Thread Angus Salkeld
On Thu, Nov 13, 2014 at 4:00 AM, Clint Byrum cl...@fewbar.com wrote:

 Excerpts from Zane Bitter's message of 2014-11-12 08:42:44 -0800:
  On 12/11/14 10:10, Clint Byrum wrote:
   Excerpts from Zane Bitter's message of 2014-11-11 13:06:17 -0800:
   On 11/11/14 13:34, Ryan Brown wrote:
   I am strongly against allowing arbitrary Javascript functions for
   complexity reasons. It's already difficult enough to get meaningful
   errors when you  up your YAML syntax.
  
   Agreed, and FWIW literally everyone that Clint has pitched the JS idea
   to thought it was crazy ;)
  
  
   So far nobody has stepped up to defend me,
 
  I'll defend you, but I can't defend the idea :)
 
   so I'll accept that maybe
   people do think it is crazy. What I'm really confused by is why we have
   a new weird ugly language like YAQL (sorry, it, like JQ, is hideous),
 
  Agreed, and appealing to its similarity with Perl or PHP (or BASIC!) is
  probably not the way to win over Python developers :D
 
   and that would somehow be less crazy than a well known mature language
   that has always been meant for embedding such as javascript.
 
  JS is a Turing-complete language, it's an entirely different kettle of
  fish to a domain-specific language that is inherently safe to interpret
  from user input. Sure, we can try to lock it down. It's a very tricky
  job to get right. (Plus it requires a new external dependency of unknown
  quality... honestly if you're going to embed a Turing-complete language,
  Python is a much more obvious choice than JS.)
 

 There's a key difference though. Python was never designed to be run
 from untrusted sources. Javascript was _from the beginning_. There are
 at least two independent javascript implementations which both have been
 designed from the ground up to run code from websites in the local
 interpreter. From the standpoint of Heat, it would be even easier to do
 this.

 Perhaps I can carve out some of that negative-1000-days of free time I
 have and I can make it a resource plugin, with the properties being code
 and references to other resources, and the attributes being the return.

   Anyway, I'd prefer YAQL over trying to get the intrinsic functions in
   HOT just right. Users will want to do things we don't expect. I say,
 let
   them, or large sections of the users will simply move on to something
   else.
 
  The other side of that argument is that users are doing one of two
  things with data they have obtained from resources in the template:
 
  1) Passing data to software deployments
  2) Passing data to other resources
 
  In case (1) they can easily transform the data into whatever format they
  want using their own scripts, running on their own server.
 
  In case (2), if it's not easy for them to just do what they want without
  having to perform this kind of manipulation, we have failed to design
  good resources. And if we give people the tools to just paper over the
  problem, we'll never hear about it so we can correct it at the source,
  just launch a thousand hard-to-maintain hacks into the world.
 


case (3) is trying to write a half useful template resource. With what we
have
this is very difficult. I think for non-trivial templates people very
quickly run
into the limitations of HOT.



 I for one would rather serve the users than ourselves, and preventing
 them from papering over the problems so they have to whine at us is a
 self-serving agenda.

 As a primary whiner about Heat for a long time, I respect a lot that
 this development team _bends over backwards_ to respond to user
 requests. It's amazing that way.

 However, I think to grow beyond open source savvy, deeply integrated
 users like me, one has to let the users solve their own problems. They'll
 know that their javascript or YAQL is debt sometimes, and they can
 come to Heat's development community with suggestions like If you had
 a coalesce function I wouldn't need to write it in javascript. But if
 you don't give them _something_, they'll just move on.


Agree, I think we need to get this done. We can't just keep ignoring users
when
they are begging for the same feature, because supposedly they are doing it
wrong.

-Angus



 Anyway, probably looking further down the road than I need to, but
 please keep an open mind for this idea, as users tend to use tools that
 solve their problem _and_ get out of their way in all other cases.

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

2014-11-12 Thread Alexis Lee
Zane Bitter said on Tue, Nov 11, 2014 at 04:06:17PM -0500:
 FWIW literally everyone that Clint has pitched the JS
 idea to thought it was crazy ;)

+1

 YAQL ... looks like line noise to me

YAML representing function call stacks (an AST) looks pretty bad to me
:)

The YAQL doc is not great at the moment but the language is not
difficult. It's pretty similar to XPath, jq and JSP EL, i.e. dotted
syntax with square brackets for attribute selection and slicing. That's
also pretty similar to Python btw.

Is there another expression language you prefer?

 From the beginning we've tried to have the absolute minimum
 number of intrinsic functions in HOT.

This is understandable but there are things I want to do (as an operator
- like first_nonnull) which I cannot. The options as I see them:

  * Add a bunch of operators and a LOT of functions
  * Add a bunch of operators, a few functions and lambda
  * Add a bunch of functions and use an expression language for
arithmetic, comparison etc
  * Use an expression language for all complex processing

 I would much prefer to resurrect the previous proposal for adding
 conditionals and then see what is still missing than to just dive
 straight in to embedding a whole other language in HOT.

Do you mean this? https://blueprints.launchpad.net/heat/+spec/intrinsics
This only provides string equality and boolean logic. It's insufficient
to implement first_nonnull for example.

I'd like to emphasise that we don't have to offer the whole YAQL stdlib
in HOT initially. We can use it just for its operator suite and a few
key functions then add others as usecases are discovered.

 On 11/11/14 13:34, Ryan Brown wrote:
 Vendored HOT
 the real issue is that it's under the control of the operator and not
 the template author.

Yep, the solution has to be through the template as that's 99% of the
user interface.


Alexis
-- 
Nova Engineer, HP Cloud.  AKA lealexis, lxsli.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

2014-11-12 Thread Angus Salkeld
On Wed, Nov 12, 2014 at 9:33 PM, Alexis Lee alex...@hp.com wrote:

 Zane Bitter said on Tue, Nov 11, 2014 at 04:06:17PM -0500:
  FWIW literally everyone that Clint has pitched the JS
  idea to thought it was crazy ;)

 +1

  YAQL ... looks like line noise to me

 YAML representing function call stacks (an AST) looks pretty bad to me
 :)

 The YAQL doc is not great at the moment but the language is not
 difficult. It's pretty similar to XPath, jq and JSP EL, i.e. dotted
 syntax with square brackets for attribute selection and slicing. That's
 also pretty similar to Python btw.

 Is there another expression language you prefer?

  From the beginning we've tried to have the absolute minimum
  number of intrinsic functions in HOT.

 This is understandable but there are things I want to do (as an operator
 - like first_nonnull) which I cannot. The options as I see them:

   * Add a bunch of operators and a LOT of functions
   * Add a bunch of operators, a few functions and lambda
   * Add a bunch of functions and use an expression language for
 arithmetic, comparison etc
   * Use an expression language for all complex processing



I think it's pretty clear we need something more powerful that adding
heaps of intrinsic functions, IMHO as long as we can prove it is safe
we should add yaql (it's nice that there would be a consistent user
experience
between these projects -mistral/heat/murano).

-Angus


  I would much prefer to resurrect the previous proposal for adding
  conditionals and then see what is still missing than to just dive
  straight in to embedding a whole other language in HOT.

 Do you mean this? https://blueprints.launchpad.net/heat/+spec/intrinsics
 This only provides string equality and boolean logic. It's insufficient
 to implement first_nonnull for example.

 I'd like to emphasise that we don't have to offer the whole YAQL stdlib
 in HOT initially. We can use it just for its operator suite and a few
 key functions then add others as usecases are discovered.


+1


  On 11/11/14 13:34, Ryan Brown wrote:
  Vendored HOT
  the real issue is that it's under the control of the operator and not
  the template author.

 Yep, the solution has to be through the template as that's 99% of the
 user interface.


 Alexis
 --
 Nova Engineer, HP Cloud.  AKA lealexis, lxsli.

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

2014-11-12 Thread Zane Bitter

On 12/11/14 06:33, Alexis Lee wrote:

I would much prefer to resurrect the previous proposal for adding
conditionals and then see what is still missing than to just dive
straight in to embedding a whole other language in HOT.

Do you mean this?https://blueprints.launchpad.net/heat/+spec/intrinsics
This only provides string equality and boolean logic. It's insufficient
to implement first_nonnull for example.


I meant this: https://review.openstack.org/#/c/84468/

- ZB

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

2014-11-12 Thread Zane Bitter

On 12/11/14 06:48, Angus Salkeld wrote:

(it's nice that there would be a consistent user experience
between these projects -mistral/heat/murano).


It's actually potentially horrible, because you introduce potential 
quoting issues when you embed mistral workbooks in Heat templates or 
pass Heat templates to Murano.


cheers,
Zane.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

2014-11-12 Thread Stan Lagun
On Wed, Nov 12, 2014 at 3:50 PM, Zane Bitter zbit...@redhat.com wrote:

 It's actually potentially horrible, because you introduce potential
 quoting issues when you embed mistral workbooks in Heat templates or pass
 Heat templates to Murano.


Could you please give an example of template with such issue?
Also note that
a. YAQL uses only single quote marks (') for string while JSON uses double
quotes ()
b. If Mistral worbook is embedded into HOT template it should not be parsed
or interpreted anyway. Otherwise consider what would happen if that
workbook would in turn have HOT snippets embedded


Sincerely yours,
Stan Lagun
Principal Software Engineer @ Mirantis

sla...@mirantis.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

2014-11-12 Thread Ryan Brown


On 11/12/2014 07:50 AM, Zane Bitter wrote:
 On 12/11/14 06:48, Angus Salkeld wrote:
 (it's nice that there would be a consistent user experience
 between these projects -mistral/heat/murano).
 
 It's actually potentially horrible, because you introduce potential
 quoting issues when you embed mistral workbooks in Heat templates or
 pass Heat templates to Murano.

I think the intention here was to say there are two paths forward.

1) Add an intrinsic for first_nonnull, and every other task.
2) Add YAQL to let users build their own expressions.

And that (1) involves users learning the Heat way of munging their data
with intrinsics, while (2) encourages them to learn YAQL, which they can
use across Heat, Mistral, and Murano. This would be a better experience
since users would (in theory) spend less time looking up the domain
language for task X.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

2014-11-12 Thread Zane Bitter

On 12/11/14 08:24, Stan Lagun wrote:


On Wed, Nov 12, 2014 at 3:50 PM, Zane Bitter zbit...@redhat.com
mailto:zbit...@redhat.com wrote:

It's actually potentially horrible, because you introduce potential
quoting issues when you embed mistral workbooks in Heat templates or
pass Heat templates to Murano.


Could you please give an example of template with such issue?


I don't have one to hand, but we'd want to prove that such an issue 
*can't* happen before implementing this.



Also note that
a. YAQL uses only single quote marks (') for string while JSON uses
double quotes ()
b. If Mistral worbook is embedded into HOT template it should not be
parsed or interpreted anyway. Otherwise consider what would happen if
that workbook would in turn have HOT snippets embedded


That's exactly the problem I'm talking about. If you have a HOT snippet 
embedded in a Mistral workbook embedded in a HOT template passed to 
Murano, can you guarantee that each piece of YAQL will be interpreted 
only in exactly the right place? Will the user be able to predict where 
that will be? My brain hurts just thinking about it.


We can protect other formats from Heat by wrapping an intrinsic function 
around the YAQL; I'm not certain that HOT would be protected from 
Mistral and Murano in the same way, because this wasn't previously a 
design goal for them.


cheers,
Zane.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

2014-11-12 Thread Clint Byrum
Excerpts from Zane Bitter's message of 2014-11-11 13:06:17 -0800:
 On 11/11/14 13:34, Ryan Brown wrote:
  I am strongly against allowing arbitrary Javascript functions for
  complexity reasons. It's already difficult enough to get meaningful
  errors when you  up your YAML syntax.
 
 Agreed, and FWIW literally everyone that Clint has pitched the JS idea 
 to thought it was crazy ;)
 

So far nobody has stepped up to defend me, so I'll accept that maybe
people do think it is crazy. What I'm really confused by is why we have
a new weird ugly language like YAQL (sorry, it, like JQ, is hideous),
and that would somehow be less crazy than a well known mature language
that has always been meant for embedding such as javascript.

Anyway, I'd prefer YAQL over trying to get the intrinsic functions in
HOT just right. Users will want to do things we don't expect. I say, let
them, or large sections of the users will simply move on to something
else.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

2014-11-12 Thread Stan Lagun
My point is that quoting problem do exists probably but it exists even
without YAQL being used anywhere.
For example consider Mistral workbook containing value: { get_attr:
[my_instance, first_address] }. get_attr in Mistral may have nothing to to
with Heat's get_attr and even if it is it may be just a HOT snippet that
cannot be evaluated at the moment. For example it may be part of a HOT
template embedded into Mistral workbook and Mistral workflow is going to
create another Heat stack using that template. Now if you embed such
workbook into HOT template (and thus get HOT embedded into Mistral embedded
into HOT :) ) you need to make sure that Heat's parser is not going to try
to parse that workbook but treats it like a black box. This is regardless
of YAQL being even exist. But if you treat workbooks like
black-boxes/arbitrary JSON you should not care if it contains YAQL or not.

As for Murano this may cause some problems in cases when someone just took
YAML HOT and copy-paste it into MuranoPL class without double-checking. But
those errors can be easily fixed and such issues do exist even with current
HOT syntax without YAQL being used at all.

What I'm really confused by is why we have
 a new weird ugly language like YAQL (sorry, it, like JQ, is hideous),


YAQL syntax is almost identical to Python's or JS if you narrow them to
one-line expressions. The only noticeable difference is that all
variable/parameter names are prefixed with '$' as in PHP/Perl which are
both popular languages. I can hardly imagine why such negligible difference
from JS or Python (languages you do like) makes language weird ugly

Sincerely yours,
Stan Lagun
Principal Software Engineer @ Mirantis

sla...@mirantis.com

On Wed, Nov 12, 2014 at 6:10 PM, Clint Byrum cl...@fewbar.com wrote:

 Excerpts from Zane Bitter's message of 2014-11-11 13:06:17 -0800:
  On 11/11/14 13:34, Ryan Brown wrote:
   I am strongly against allowing arbitrary Javascript functions for
   complexity reasons. It's already difficult enough to get meaningful
   errors when you  up your YAML syntax.
 
  Agreed, and FWIW literally everyone that Clint has pitched the JS idea
  to thought it was crazy ;)
 

 So far nobody has stepped up to defend me, so I'll accept that maybe
 people do think it is crazy. What I'm really confused by is why we have
 a new weird ugly language like YAQL (sorry, it, like JQ, is hideous),
 and that would somehow be less crazy than a well known mature language
 that has always been meant for embedding such as javascript.

 Anyway, I'd prefer YAQL over trying to get the intrinsic functions in
 HOT just right. Users will want to do things we don't expect. I say, let
 them, or large sections of the users will simply move on to something
 else.

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

2014-11-12 Thread Zane Bitter

On 12/11/14 10:10, Clint Byrum wrote:

Excerpts from Zane Bitter's message of 2014-11-11 13:06:17 -0800:

On 11/11/14 13:34, Ryan Brown wrote:

I am strongly against allowing arbitrary Javascript functions for
complexity reasons. It's already difficult enough to get meaningful
errors when you  up your YAML syntax.


Agreed, and FWIW literally everyone that Clint has pitched the JS idea
to thought it was crazy ;)



So far nobody has stepped up to defend me,


I'll defend you, but I can't defend the idea :)


so I'll accept that maybe
people do think it is crazy. What I'm really confused by is why we have
a new weird ugly language like YAQL (sorry, it, like JQ, is hideous),


Agreed, and appealing to its similarity with Perl or PHP (or BASIC!) is 
probably not the way to win over Python developers :D



and that would somehow be less crazy than a well known mature language
that has always been meant for embedding such as javascript.


JS is a Turing-complete language, it's an entirely different kettle of 
fish to a domain-specific language that is inherently safe to interpret 
from user input. Sure, we can try to lock it down. It's a very tricky 
job to get right. (Plus it requires a new external dependency of unknown 
quality... honestly if you're going to embed a Turing-complete language, 
Python is a much more obvious choice than JS.)



Anyway, I'd prefer YAQL over trying to get the intrinsic functions in
HOT just right. Users will want to do things we don't expect. I say, let
them, or large sections of the users will simply move on to something
else.


The other side of that argument is that users are doing one of two 
things with data they have obtained from resources in the template:


1) Passing data to software deployments
2) Passing data to other resources

In case (1) they can easily transform the data into whatever format they 
want using their own scripts, running on their own server.


In case (2), if it's not easy for them to just do what they want without 
having to perform this kind of manipulation, we have failed to design 
good resources. And if we give people the tools to just paper over the 
problem, we'll never hear about it so we can correct it at the source, 
just launch a thousand hard-to-maintain hacks into the world.


cheers,
Zane.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

2014-11-12 Thread Randall Burt
On Nov 12, 2014, at 10:42 AM, Zane Bitter zbit...@redhat.com
 wrote:

 On 12/11/14 10:10, Clint Byrum wrote:
 Excerpts from Zane Bitter's message of 2014-11-11 13:06:17 -0800:
 On 11/11/14 13:34, Ryan Brown wrote:
 I am strongly against allowing arbitrary Javascript functions for
 complexity reasons. It's already difficult enough to get meaningful
 errors when you  up your YAML syntax.
 
 Agreed, and FWIW literally everyone that Clint has pitched the JS idea
 to thought it was crazy ;)
 
 
 So far nobody has stepped up to defend me,
 
 I'll defend you, but I can't defend the idea :)
 
 so I'll accept that maybe
 people do think it is crazy. What I'm really confused by is why we have
 a new weird ugly language like YAQL (sorry, it, like JQ, is hideous),
 
 Agreed, and appealing to its similarity with Perl or PHP (or BASIC!) is 
 probably not the way to win over Python developers :D
 
 and that would somehow be less crazy than a well known mature language
 that has always been meant for embedding such as javascript.
 
 JS is a Turing-complete language, it's an entirely different kettle of fish 
 to a domain-specific language that is inherently safe to interpret from user 
 input. Sure, we can try to lock it down. It's a very tricky job to get right. 
 (Plus it requires a new external dependency of unknown quality... honestly if 
 you're going to embed a Turing-complete language, Python is a much more 
 obvious choice than JS.)
 
 Anyway, I'd prefer YAQL over trying to get the intrinsic functions in
 HOT just right. Users will want to do things we don't expect. I say, let
 them, or large sections of the users will simply move on to something
 else.
 
 The other side of that argument is that users are doing one of two things 
 with data they have obtained from resources in the template:
 
 1) Passing data to software deployments
 2) Passing data to other resources
 
 In case (1) they can easily transform the data into whatever format they want 
 using their own scripts, running on their own server.
 
 In case (2), if it's not easy for them to just do what they want without 
 having to perform this kind of manipulation, we have failed to design good 
 resources. And if we give people the tools to just paper over the problem, 
 we'll never hear about it so we can correct it at the source, just launch a 
 thousand hard-to-maintain hacks into the world.

I disagree with this last bit. Having some manner of data manipulation facility 
as part of the template language doesn't mean the resources are badly designed 
or implemented, it just means that users have constraints that we don't 
directly address. Having something more general purpose (whatever it may be) 
seems like a longer term and more maintainable solution to me than eternal tail 
chasing where 1000 specific intrinsic functions bloom. IMO, intrinsics have a 
place, but supporting something like YAQL (or whatever) seems like a general 
solution to a lot of common problems our users come up against.

As for hard-to-maintain hacks, isn't that what drives a lot of our users to 
ask for these sorts of solutions in the first place? If doing something via 
YAQL is perceived by a user as some sub-optimal hack for their use case, I see 
no reason for that to be any less visible than any other pain point or 
usability issue we have.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

2014-11-12 Thread Clint Byrum
Excerpts from Zane Bitter's message of 2014-11-12 08:42:44 -0800:
 On 12/11/14 10:10, Clint Byrum wrote:
  Excerpts from Zane Bitter's message of 2014-11-11 13:06:17 -0800:
  On 11/11/14 13:34, Ryan Brown wrote:
  I am strongly against allowing arbitrary Javascript functions for
  complexity reasons. It's already difficult enough to get meaningful
  errors when you  up your YAML syntax.
 
  Agreed, and FWIW literally everyone that Clint has pitched the JS idea
  to thought it was crazy ;)
 
 
  So far nobody has stepped up to defend me,
 
 I'll defend you, but I can't defend the idea :)
 
  so I'll accept that maybe
  people do think it is crazy. What I'm really confused by is why we have
  a new weird ugly language like YAQL (sorry, it, like JQ, is hideous),
 
 Agreed, and appealing to its similarity with Perl or PHP (or BASIC!) is 
 probably not the way to win over Python developers :D
 
  and that would somehow be less crazy than a well known mature language
  that has always been meant for embedding such as javascript.
 
 JS is a Turing-complete language, it's an entirely different kettle of 
 fish to a domain-specific language that is inherently safe to interpret 
 from user input. Sure, we can try to lock it down. It's a very tricky 
 job to get right. (Plus it requires a new external dependency of unknown 
 quality... honestly if you're going to embed a Turing-complete language, 
 Python is a much more obvious choice than JS.)
 

There's a key difference though. Python was never designed to be run
from untrusted sources. Javascript was _from the beginning_. There are
at least two independent javascript implementations which both have been
designed from the ground up to run code from websites in the local
interpreter. From the standpoint of Heat, it would be even easier to do
this.

Perhaps I can carve out some of that negative-1000-days of free time I
have and I can make it a resource plugin, with the properties being code
and references to other resources, and the attributes being the return.

  Anyway, I'd prefer YAQL over trying to get the intrinsic functions in
  HOT just right. Users will want to do things we don't expect. I say, let
  them, or large sections of the users will simply move on to something
  else.
 
 The other side of that argument is that users are doing one of two 
 things with data they have obtained from resources in the template:
 
 1) Passing data to software deployments
 2) Passing data to other resources
 
 In case (1) they can easily transform the data into whatever format they 
 want using their own scripts, running on their own server.
 
 In case (2), if it's not easy for them to just do what they want without 
 having to perform this kind of manipulation, we have failed to design 
 good resources. And if we give people the tools to just paper over the 
 problem, we'll never hear about it so we can correct it at the source, 
 just launch a thousand hard-to-maintain hacks into the world.
 

I for one would rather serve the users than ourselves, and preventing
them from papering over the problems so they have to whine at us is a
self-serving agenda.

As a primary whiner about Heat for a long time, I respect a lot that
this development team _bends over backwards_ to respond to user
requests. It's amazing that way.

However, I think to grow beyond open source savvy, deeply integrated
users like me, one has to let the users solve their own problems. They'll
know that their javascript or YAQL is debt sometimes, and they can
come to Heat's development community with suggestions like If you had
a coalesce function I wouldn't need to write it in javascript. But if
you don't give them _something_, they'll just move on.

Anyway, probably looking further down the road than I need to, but
please keep an open mind for this idea, as users tend to use tools that
solve their problem _and_ get out of their way in all other cases.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

2014-11-11 Thread Randall Burt
I like this approach and seems to have greater utility beyond the original 
proposal. I'm not fussed on YAQL or straight up JSONPath, but something along 
those lines seems to make sense.

On Nov 11, 2014, at 9:35 AM, Alexis Lee alex...@hp.com
 wrote:

 Alexis Lee said on Mon, Nov 10, 2014 at 05:34:13PM +:
 How about we support YAQL expressions? https://github.com/ativelkov/yaql
 Plus some HOFs (higher-order functions) like cond, map, filter, foldleft
 etc?
 
 We could also use YAQL to provide the HOFs.
 
 Here's first_nonnull:
 
  config:
Fn::Select
  - 0
  filter:
- yaql: $.0 != null
- item1
- itemN
 
  config:
yaql: $[$ != null][0]
- item1
- itemN
 
 This approach requires less change to Heat, at the price of learning
 more YAQL.
 
 
 Alexis
 -- 
 Nova Engineer, HP Cloud.  AKA lealexis, lxsli.
 
 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

2014-11-11 Thread Zane Bitter

On 10/11/14 12:34, Alexis Lee wrote:

Zane Bitter said on Fri, Nov 07, 2014 at 12:35:09AM +0100:

Crazy thought: why not just implement conditionals? We had a
proto-spec for them started at one point...


I didn't know that was on the table :)

How about we support YAQL expressions? https://github.com/ativelkov/yaql
Plus some HOFs (higher-order functions) like cond, map, filter, foldleft
etc?

Here's first_nonnull:

   config:
 Fn::Select
   - 0
   filter:
 - yaql: $.0 != null
 - item1
 - itemN

Making the 'yaql' function eponymous means we can easily plug other
expression languages in later if we choose.

We can restrict the functions available to YAQL expressions to avoid a
halting problem.

Each HOF will know how to populate the context for subordinate YAQL
expressions. When used outside a HOF, only add a default context of
parameters and input_values.

Is this totally crazy or worth spec-ifying?


The first one.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

2014-11-11 Thread Clint Byrum
Excerpts from Alexis Lee's message of 2014-11-10 09:34:13 -0800:
 Zane Bitter said on Fri, Nov 07, 2014 at 12:35:09AM +0100:
  Crazy thought: why not just implement conditionals? We had a
  proto-spec for them started at one point...
 
 I didn't know that was on the table :)
 
 How about we support YAQL expressions? https://github.com/ativelkov/yaql
 Plus some HOFs (higher-order functions) like cond, map, filter, foldleft
 etc?
 
 Here's first_nonnull:
 
   config:
 Fn::Select
   - 0
   filter:
 - yaql: $.0 != null
 - item1
 - itemN
 
 Making the 'yaql' function eponymous means we can easily plug other
 expression languages in later if we choose.
 

Cool. I think this aligns nicely with my suggestion at the summit that
we also allow writing functions in javascript. This would allow people
to write their own first_nonnull/coalesce until we have a chance to add
it:

  config:
embed:
  lang: javascript
  args: - item1
- item2
  code: |
for (arg in args) {
  if (arg !== null) {
return arg
  }
}
return null;

I'd really love to have this functionality but I doubt I'll have time to
spec and land it. Does anybody else think this functionality would be a
good way to allow template authors flexibility that they desire?

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

2014-11-11 Thread Ryan Brown
On 11/11/2014 01:12 PM, Clint Byrum wrote:
 Excerpts from Alexis Lee's message of 2014-11-10 09:34:13 -0800:
 Zane Bitter said on Fri, Nov 07, 2014 at 12:35:09AM +0100:
 Crazy thought: why not just implement conditionals? We had a
 proto-spec for them started at one point...

 I didn't know that was on the table :)

 How about we support YAQL expressions? https://github.com/ativelkov/yaql
 Plus some HOFs (higher-order functions) like cond, map, filter, foldleft
 etc?

 Here's first_nonnull:

   config:
 Fn::Select
   - 0
   filter:
 - yaql: $.0 != null
 - item1
 - itemN

 Making the 'yaql' function eponymous means we can easily plug other
 expression languages in later if we choose.

Adding yaql (or similar, e.g. jsonpointer) seems like it would be a good
idea, and shouldn't add more complexity than it's worth.

 Cool. I think this aligns nicely with my suggestion at the summit that
 we also allow writing functions in javascript. This would allow people
 to write their own first_nonnull/coalesce until we have a chance to add
 it:
 
   config:
 embed:
   lang: javascript
   args: - item1
 - item2
   code: |
 for (arg in args) {
   if (arg !== null) {
 return arg
   }
 }
 return null;
 
 I'd really love to have this functionality but I doubt I'll have time to
 spec and land it. Does anybody else think this functionality would be a
 good way to allow template authors flexibility that they desire?

I am strongly against allowing arbitrary Javascript functions for
complexity reasons. It's already difficult enough to get meaningful
errors when you  up your YAML syntax.

AIUI the functionality many users would look to use Javascript
embed-ability for would be better served either by something like yaql,
or by making vendored HOT functions possible.

Vendored HOT would look something like X-Vendor::YourNamespace and
functions could be managed similarly to resource plugins (stevedore).
It's a very rough idea, but I like it much better than adding Javascript.

-- 
Ryan Brown / Software Engineer, Openstack / Red Hat, Inc.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

2014-11-11 Thread Zane Bitter

On 11/11/14 13:34, Ryan Brown wrote:

I am strongly against allowing arbitrary Javascript functions for
complexity reasons. It's already difficult enough to get meaningful
errors when you  up your YAML syntax.


Agreed, and FWIW literally everyone that Clint has pitched the JS idea 
to thought it was crazy ;)


I don't really like YAQL either though. It looks like line noise to me. 
From the beginning we've tried to have the absolute minimum number of 
intrinsic functions in HOT. I would much prefer to resurrect the 
previous proposal for adding conditionals and then see what is still 
missing than to just dive straight in to embedding a whole other 
language in HOT.



AIUI the functionality many users would look to use Javascript
embed-ability for would be better served either by something like yaql,
or by making vendored HOT functions possible.

Vendored HOT would look something like X-Vendor::YourNamespace and
functions could be managed similarly to resource plugins (stevedore).
It's a very rough idea, but I like it much better than adding Javascript.


That already existed at one point. We ripped out the plugin part during 
Juno in favour of tying the intrinsic functions available to a template 
format, so that vendoring HOT with new intrinsic functions also requires 
(or at least suggests) changing the template format version string, to 
allow Heat can detect immediately when you are using an unsupported 
template format.


That said it still basically works, but the real issue is that it's 
under the control of the operator and not the template author.


cheers,
Zane.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

2014-11-11 Thread Angus Salkeld
On Wed, Nov 12, 2014 at 1:35 AM, Alexis Lee alex...@hp.com wrote:

 Alexis Lee said on Mon, Nov 10, 2014 at 05:34:13PM +:
  How about we support YAQL expressions? https://github.com/ativelkov/yaql
  Plus some HOFs (higher-order functions) like cond, map, filter, foldleft
  etc?

 We could also use YAQL to provide the HOFs.

  Here's first_nonnull:
 
config:
  Fn::Select
- 0
filter:
  - yaql: $.0 != null
  - item1
  - itemN

   config:
 yaql: $[$ != null][0]
 - item1
 - itemN

 This approach requires less change to Heat, at the price of learning
 more YAQL.


+1 to YAQL


-Angus




 Alexis
 --
 Nova Engineer, HP Cloud.  AKA lealexis, lxsli.

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

2014-11-11 Thread Stan Lagun
For those who don't know 100% of guys behind YAQL are also active OpenStack
contributors. During (early) Kilo cycle we plan to release YAQL version 1.0
on stackforge. This release is going to fix some flaws in early versions,
add some more flexibility and have very high UT coverage. There are at
least 2 projects in OpenStack that already make heavy use of YAQL: Mistral
and Murano. If Heat team decides to use YAQL as well we will be glad to
help integrate it smoothly with HOT as well as make sure that YAQL 1.0 is
aligned with Heat use cases.

Sincerely yours,
Stan Lagun
Principal Software Engineer @ Mirantis

sla...@mirantis.com

On Wed, Nov 12, 2014 at 12:32 AM, Angus Salkeld asalk...@mirantis.com
wrote:



 On Wed, Nov 12, 2014 at 1:35 AM, Alexis Lee alex...@hp.com wrote:

 Alexis Lee said on Mon, Nov 10, 2014 at 05:34:13PM +:
  How about we support YAQL expressions?
 https://github.com/ativelkov/yaql
  Plus some HOFs (higher-order functions) like cond, map, filter, foldleft
  etc?

 We could also use YAQL to provide the HOFs.

  Here's first_nonnull:
 
config:
  Fn::Select
- 0
filter:
  - yaql: $.0 != null
  - item1
  - itemN

   config:
 yaql: $[$ != null][0]
 - item1
 - itemN

 This approach requires less change to Heat, at the price of learning
 more YAQL.


 +1 to YAQL


 -Angus




 Alexis
 --
 Nova Engineer, HP Cloud.  AKA lealexis, lxsli.

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Heat] Conditionals, was: New function: first_nonnull

2014-11-10 Thread Alexis Lee
Zane Bitter said on Fri, Nov 07, 2014 at 12:35:09AM +0100:
 Crazy thought: why not just implement conditionals? We had a
 proto-spec for them started at one point...

I didn't know that was on the table :)

How about we support YAQL expressions? https://github.com/ativelkov/yaql
Plus some HOFs (higher-order functions) like cond, map, filter, foldleft
etc?

Here's first_nonnull:

  config:
Fn::Select
  - 0
  filter:
- yaql: $.0 != null
- item1
- itemN

Making the 'yaql' function eponymous means we can easily plug other
expression languages in later if we choose.

We can restrict the functions available to YAQL expressions to avoid a
halting problem.

Each HOF will know how to populate the context for subordinate YAQL
expressions. When used outside a HOF, only add a default context of
parameters and input_values.

Is this totally crazy or worth spec-ifying?


Alexis
-- 
Nova Engineer, HP Cloud.  AKA lealexis, lxsli.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev