Hell All,
Was I too hasty to merge the PR -
https://github.com/apache/brooklyn-server/pull/1358?
<https://github.com/apache/brooklyn-server/pull/1358?>
We agreed to split the some of the work with Alex, and I did not see why it
could not be merged to continue with what I’m working on.
Regards,
Mykola
> On 9 Sep 2022, at 20:17, Geoff Macartney <geom...@apache.org> wrote:
>
> Hi Alex,
>
> Thanks for the link, will have a look at the PR.
>
> As for the new shorthand proposal, +1 from me. I think it will be a
> majority of cases when things *are* simple - setting sensors, etc.
> Your example above goes from eight lines of markup for the steps to
> two, I think that's a compelling argument in itself in favour of the
> proposal.
>
> Cheers
> Geoff
>
> On Thu, 8 Sept 2022 at 09:35, Alex Heneveld <a...@cloudsoft.io> wrote:
>>
>> Hi team,
>>
>> An initial PR with a few types and the ability to define an effector is
>> available [1].
>>
>> This is enough for the next steps to be parallelized, e.g. new steps
>> added. The proposal has been updated with a work plan / list of tasks
>> [2]. Any volunteers to help with some of the upcoming tasks let me know.
>>
>> Finally I've been thinking about the "shorthand syntax" and how to bring us
>> closer to Peter's proposal of a DSL. The original proposal allowed instead
>> of a map e.g.
>>
>> step_sleep:
>> type: sleep
>> duration: 5s
>>
>> or
>>
>> step_update_service_up:
>> type: set-sensor
>> sensor:
>> name: service.isUp
>> type: boolean
>> value: true
>>
>> being able to use a shorthand _map_ with a single key being the type, and
>> value interpreted by that type, so in the OLD SHORTHAND PROPOSAL the above
>> could be written:
>>
>> step_sleep:
>> sleep: 5s
>>
>> step_update_service_up:
>> set-sensor: service.isUp = true
>>
>> Having played with syntaxes a bit I wonder if we should instead say the
>> shorthand DSL kicks in when the step _body_ is a string (instead of a
>> single-key map), and the first word of the string being the type, and the
>> remainder interpreted by the type, and we allow it to be a bit more
>> ambitious.
>>
>> Concretely this NEW SHORTHAND PROPOSAL would look something like:
>>
>> step_sleep: sleep 5s
>> step_update_service_up: set-sensor service.isUp = true
>> # also supporting a type, ie `set-sensor [TYPE] NAME = VALUE`, eg
>> step_update_service_up: set-sensor boolean service.isUp = true
>>
>> You would still need the full map syntax whenever defining flow logic -- eg
>> condition, next, retry, or timeout -- or any property not supported by the
>> shorthand syntax. But for the (majority?) simple cases the expression
>> would be very concise. In most cases I think it would feel like a DSL but
>> has the virtue of a very clear translation to the actual workflow model and
>> the underlying (YAML) model needed for resumption and UI.
>>
>> As a final example, the example used at the start of the proposal
>> (simplified a little -- removing on-error retry and env map as those
>> wouldn't be supported by shorthand):
>>
>> brooklyn.initializers:
>> - type: workflow-effector
>> name: run-spark-on-gcp
>> steps:
>> 1:
>> type: container
>> image: my/google-cloud
>> command: gcloud dataproc jobs submit spark
>> --BUCKET=gs://$brooklyn:config("bucket")
>> 2:
>> type: set-sensor
>> sensor: spark-output
>> value: ${1.stdout}
>>
>> Could be written in this shorthand as follows:
>>
>> steps:
>> 1: container my/google-cloud command "gcloud dataproc jobs submit spark
>> --BUCKET=gs://${entity.config.bucket}"
>> 2: set-sensor spark-output ${1.stdout}
>>
>> Thoughts?
>>
>> Best
>> Alex
>>
>>
>> [1] https://github.com/apache/brooklyn-server/pull/1358
>> [2]
>> https://docs.google.com/document/d/1u02Bi6sS8Fkf1s7UzRRMnvLhA477bqcyxGa0nJesqkI/edit#heading=h.gbadaqa2yql6
>>
>>
>> On Wed, 7 Sept 2022 at 09:58, Alex Heneveld <a...@cloudsoft.io> wrote:
>>
>>> Hi Peter,
>>>
>>> Yes - thanks for the extra details. I did take your suggestion to be a
>>> procedural DSL not YAML, per the illustration at [1] (second code block).
>>> Probably where I was confusing was in saying that unlike DSLs which just
>>> run (and where the execution can be delegated to eg java/groovy/ruby), here
>>> we need to understand and display, store and resume the workflow progress.
>>> So I think it needs to be compiled to some representation that is well
>>> described and that new Apache Brooklyn code can reason about, both in the
>>> UI (JS) and backend (Java). Parsing a DSL is much harder than using YAML
>>> for this "reasonable" representation (as in we can reason _about_ it :) ),
>>> because we already have good backend processing, persistence,
>>> serialization; and frontend processing and visualization support for
>>> YAML-based models. So I think we almost definitely want a well-described
>>> declarative YAML model of the workflow.
>>>
>>> We might *also* want a Workflow DSL because I agree with you a DSL would
>>> be nicer for a user to write (if writing by hand; although if composing
>>> visually a drag-and-drop to YAML is probably easier). However it should
>>> probably get "compiled" into a Workflow YAML. So I'm suggesting we do the
>>> workflow YAML at this stage, and a DSL that compiles into that YAML can be
>>> designed later. (Designing a good DSL and parser and reason-about-able
>>> representation is a big task, so being able to separate it feels good too!)
>>>
>>> Best
>>> Alex
>>>
>>> [1]
>>> https://docs.google.com/document/d/1u02Bi6sS8Fkf1s7UzRRMnvLhA477bqcyxGa0nJesqkI/edit#heading=h.75wm48pjvx0h
>>>
>>>
>>> On Fri, 2 Sept 2022 at 20:17, Geoff Macartney <geoff.macart...@gmail.com>
>>> wrote:
>>>
>>>> Hi Peter,
>>>>
>>>> Thanks for such a detailed writeup of how you see this working. I fear
>>>> I've too little experience with this sort of thing to be able to say
>>>> anything very useful about it. My thought on the matter would be,
>>>> let's get started with the yaml based approach and see how it goes. I
>>>> think that experience would then give us a much better feel for what a
>>>> really nice and usable DSL for workflows would look like (probably to
>>>> address all the pain points of the yaml approach! :-) The outline
>>>> above will then be a good starting point, I'm sure.
>>>>
>>>> Cheers
>>>> Geoff
>>>>
>>>> On Thu, 1 Sept 2022 at 21:26, Peter Abramowitsch
>>>> <pabramowit...@gmail.com> wrote:
>>>>>
>>>>> Hi All
>>>>> I just wanted to clarify something in my comment the other day about
>>>> DSLs
>>>>> since I see that the acronym was also used in Alex's original document.
>>>>> Unless I misunderstood, Alex was proposing to create a DSL for Brooklyn
>>>>> using yaml as syntax and writing a code layer to translate between that
>>>>> syntax and underlying APIs which are presumably all in Java.
>>>>>
>>>>> What I was suggesting was a DSL written directly in Java (I guess)
>>>> whose
>>>>> syntax would be that language, but whose grammar would be keywords that
>>>>> were also Java functions. Some of these functions would be pre-defined
>>>> in
>>>>> the DSL, while others could be defined by the user and could use other
>>>>> functions of the DSL. The result would be turned into a JAR file (or
>>>>> equivalent in another platform) But during the compile phase, it
>>>> would be
>>>>> checked for errors, and it could be debugged line by line either
>>>> invoking
>>>>> live functionality or using a library of mock versions of the Brooklyn
>>>> API.
>>>>>
>>>>> In this 'native' DSL one could provide different types of workflow
>>>>> constructs as functions (In the BaseClass), taking function names as
>>>> method
>>>>> pointers, or using Lambdas. It would be a lot easier in Ruby or Python
>>>>>
>>>>> // linear
>>>>> brooklynRun(NamedTaskMethod, NamedTaskMethod)
>>>>>
>>>>> // chained
>>>>> TaskMethodA()TaskMethodB().
>>>>>
>>>>> // asynchronous
>>>>> brooklynJoin(NamedTaskMethod, NamedTaskMethod,...)
>>>>>
>>>>> // conditional
>>>>> brooklynRunIf(NamedTaskMethod, NamedConditionMethod,...)
>>>>>
>>>>> // iterative
>>>>> brooklynRunWhile(NamedTaskMethod, NamedConditionMethod,...)
>>>>> brooklynRunUntil(NamedTaskMethod, NamedConditionMethod,...)
>>>>>
>>>>> // there could even be a utility to implement legacy syntax (this of
>>>> course
>>>>> would require the extra code layer I was trying to avoid)
>>>>> runYaml(Path)
>>>>>
>>>>> A basic class structure might be
>>>>>
>>>>> // where BrooklynRecipeBase implements the utility functions including,
>>>>> among others Join, Run, If, While, Until mentioned above
>>>>> // and the BrooklynWorkflowInterface would dictate the functional
>>>>> requirements for the mandatory aspects of the Recipe.
>>>>> class MyRecipe extends BrooklynRecipeBase implements,
>>>>> BrooklynWorkflowInterface
>>>>> {
>>>>> Initialize()
>>>>> createContext() - spin up resources
>>>>> workflow() - the main launch sequence using aspects of the DSL
>>>>> monitoring() - an asynchronous workflow used to manage sensor output or
>>>> for
>>>>> whatever needs to be done while the "orchestra" is plating
>>>>> shutdownHook() - called whenever shutdown is happening
>>>>> }
>>>>>
>>>>> For those who don't like the smell of Java, the source file could just
>>>> be
>>>>> the contents, which would then be injected into the class framing code
>>>>> before compilation.
>>>>>
>>>>> These are just ideas. I'm not familiar enough with Brooklyn in its
>>>> current
>>>>> implementation to be able to create realistic pseudocode.
>>>>>
>>>>> Peter
>>>>>
>>>>> On Thu, Sep 1, 2022 at 9:24 AM Geoff Macartney <
>>>> geoff.macart...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Alex,
>>>>>>
>>>>>> That's great, I'll be excited to hear all about it. 7th September
>>>>>> suits me fine; I would probably prefer 4.00 p.m. over 11.00.
>>>>>>
>>>>>> Cheers
>>>>>> Geoff
>>>>>>
>>>>>> On Thu, 1 Sept 2022 at 12:41, Alex Heneveld <a...@cloudsoft.io>
>>>> wrote:
>>>>>>>
>>>>>>> Thanks for the excellent feedback Geoff and yes there are some very
>>>> cool
>>>>>> and exciting things added recently -- containers, conditions, and
>>>> terraform
>>>>>> and kubernetes support, all of which make writing complex blueprints
>>>> much
>>>>>> easier.
>>>>>>>
>>>>>>> I'd love to host a session to showcase these.
>>>>>>>
>>>>>>> How does Wed 7 Sept sound? I could do 11am UK or 4pm UK --
>>>> depending
>>>>>> what time suits for people who are interested. Please RSVP and
>>>> indicate
>>>>>> your time preference!
>>>>>>>
>>>>>>> Best
>>>>>>> Alex
>>>>>>>
>>>>>>>
>>>>>>> On Wed, 31 Aug 2022 at 22:17, Geoff Macartney <
>>>> geoff.macart...@gmail.com>
>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi Alex,
>>>>>>>>
>>>>>>>> Another thought occurred to me when reading that workflow
>>>> proposal. You
>>>>>> wrote
>>>>>>>>
>>>>>>>> "and with the recent support for container-based tasks and
>>>> declarative
>>>>>>>> conditions, we have taken big steps towards enabling YAML
>>>> authorship"
>>>>>>>>
>>>>>>>> Unfortunately over the past while I haven't been able to keep up as
>>>>>>>> closely as I would like with developments in Brooklyn. I'm just
>>>>>>>> wondering if it might be possible to get together some time, on
>>>> Google
>>>>>>>> Meet or Zoom or whatnot, if you or a colleague could spare half an
>>>>>>>> hour to demo some of these recent developments? But don't worry
>>>> about
>>>>>>>> it if you're too busy at present.
>>>>>>>>
>>>>>>>> Adding dev@ to this in CC for the sake of Openness. Others might
>>>> also
>>>>>>>> be interested!
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>> Geoff
>>>>>>
>>>>
>>>
