Yes, Geoff, I myself have spent a couple of hours reviewing it. I’m am ready to continue developing workflows further, and I would not delay it if there are no strong objections for workflows. It looks like a good addition to the core, moreover, looks like there is a demand for it.
Please leave comments in the PR that is merged and I address them, respectively. Mykola > On 12 Sep 2022, at 09:59, Geoff Macartney <geoff.macart...@gmail.com> wrote: > > Hi Mykola, > > I'd have thought it would be worth giving it some time to review, > given that it is such a significant change. I'm personally of the > opinion that significant changes like this shouldn't be merged without > review. I was hoping to do some review on it this week myself, though > I don't really get time to work on Brooklyn these days, so I was also > hoping that others would review it who are still close to the code. > Did you get a chance to look through it yourself? > > What do you all think? I guess if it is in place and doesn't break > anything then it's OK? > > Geoff > > > > On Mon, 12 Sept 2022 at 09:04, Mykola Mandra > <mykola.man...@cloudsoftcorp.com> wrote: >> >> Hell All, >> >> Was I too hasty to merge the PR - >> https://github.com/apache/brooklyn-server/pull/1358? >> <https://github.com/apache/brooklyn-server/pull/1358?> >> We agreed to split the some of the work with Alex, and I did not see why it >> could not be merged to continue with what I’m working on. >> >> Regards, >> Mykola >> >>> On 9 Sep 2022, at 20:17, Geoff Macartney <geom...@apache.org> wrote: >>> >>> Hi Alex, >>> >>> Thanks for the link, will have a look at the PR. >>> >>> As for the new shorthand proposal, +1 from me. I think it will be a >>> majority of cases when things *are* simple - setting sensors, etc. >>> Your example above goes from eight lines of markup for the steps to >>> two, I think that's a compelling argument in itself in favour of the >>> proposal. >>> >>> Cheers >>> Geoff >>> >>> On Thu, 8 Sept 2022 at 09:35, Alex Heneveld <a...@cloudsoft.io> wrote: >>>> >>>> Hi team, >>>> >>>> An initial PR with a few types and the ability to define an effector is >>>> available [1]. >>>> >>>> This is enough for the next steps to be parallelized, e.g. new steps >>>> added. The proposal has been updated with a work plan / list of tasks >>>> [2]. Any volunteers to help with some of the upcoming tasks let me know. >>>> >>>> Finally I've been thinking about the "shorthand syntax" and how to bring us >>>> closer to Peter's proposal of a DSL. The original proposal allowed instead >>>> of a map e.g. >>>> >>>> step_sleep: >>>> type: sleep >>>> duration: 5s >>>> >>>> or >>>> >>>> step_update_service_up: >>>> type: set-sensor >>>> sensor: >>>> name: service.isUp >>>> type: boolean >>>> value: true >>>> >>>> being able to use a shorthand _map_ with a single key being the type, and >>>> value interpreted by that type, so in the OLD SHORTHAND PROPOSAL the above >>>> could be written: >>>> >>>> step_sleep: >>>> sleep: 5s >>>> >>>> step_update_service_up: >>>> set-sensor: service.isUp = true >>>> >>>> Having played with syntaxes a bit I wonder if we should instead say the >>>> shorthand DSL kicks in when the step _body_ is a string (instead of a >>>> single-key map), and the first word of the string being the type, and the >>>> remainder interpreted by the type, and we allow it to be a bit more >>>> ambitious. >>>> >>>> Concretely this NEW SHORTHAND PROPOSAL would look something like: >>>> >>>> step_sleep: sleep 5s >>>> step_update_service_up: set-sensor service.isUp = true >>>> # also supporting a type, ie `set-sensor [TYPE] NAME = VALUE`, eg >>>> step_update_service_up: set-sensor boolean service.isUp = true >>>> >>>> You would still need the full map syntax whenever defining flow logic -- eg >>>> condition, next, retry, or timeout -- or any property not supported by the >>>> shorthand syntax. But for the (majority?) simple cases the expression >>>> would be very concise. In most cases I think it would feel like a DSL but >>>> has the virtue of a very clear translation to the actual workflow model and >>>> the underlying (YAML) model needed for resumption and UI. >>>> >>>> As a final example, the example used at the start of the proposal >>>> (simplified a little -- removing on-error retry and env map as those >>>> wouldn't be supported by shorthand): >>>> >>>> brooklyn.initializers: >>>> - type: workflow-effector >>>> name: run-spark-on-gcp >>>> steps: >>>> 1: >>>> type: container >>>> image: my/google-cloud >>>> command: gcloud dataproc jobs submit spark >>>> --BUCKET=gs://$brooklyn:config("bucket") >>>> 2: >>>> type: set-sensor >>>> sensor: spark-output >>>> value: ${1.stdout} >>>> >>>> Could be written in this shorthand as follows: >>>> >>>> steps: >>>> 1: container my/google-cloud command "gcloud dataproc jobs submit spark >>>> --BUCKET=gs://${entity.config.bucket}" >>>> 2: set-sensor spark-output ${1.stdout} >>>> >>>> Thoughts? >>>> >>>> Best >>>> Alex >>>> >>>> >>>> [1] https://github.com/apache/brooklyn-server/pull/1358 >>>> [2] >>>> https://docs.google.com/document/d/1u02Bi6sS8Fkf1s7UzRRMnvLhA477bqcyxGa0nJesqkI/edit#heading=h.gbadaqa2yql6 >>>> >>>> >>>> On Wed, 7 Sept 2022 at 09:58, Alex Heneveld <a...@cloudsoft.io> wrote: >>>> >>>>> Hi Peter, >>>>> >>>>> Yes - thanks for the extra details. I did take your suggestion to be a >>>>> procedural DSL not YAML, per the illustration at [1] (second code block). >>>>> Probably where I was confusing was in saying that unlike DSLs which just >>>>> run (and where the execution can be delegated to eg java/groovy/ruby), >>>>> here >>>>> we need to understand and display, store and resume the workflow progress. >>>>> So I think it needs to be compiled to some representation that is well >>>>> described and that new Apache Brooklyn code can reason about, both in the >>>>> UI (JS) and backend (Java). Parsing a DSL is much harder than using YAML >>>>> for this "reasonable" representation (as in we can reason _about_ it :) ), >>>>> because we already have good backend processing, persistence, >>>>> serialization; and frontend processing and visualization support for >>>>> YAML-based models. So I think we almost definitely want a well-described >>>>> declarative YAML model of the workflow. >>>>> >>>>> We might *also* want a Workflow DSL because I agree with you a DSL would >>>>> be nicer for a user to write (if writing by hand; although if composing >>>>> visually a drag-and-drop to YAML is probably easier). However it should >>>>> probably get "compiled" into a Workflow YAML. So I'm suggesting we do the >>>>> workflow YAML at this stage, and a DSL that compiles into that YAML can be >>>>> designed later. (Designing a good DSL and parser and reason-about-able >>>>> representation is a big task, so being able to separate it feels good >>>>> too!) >>>>> >>>>> Best >>>>> Alex >>>>> >>>>> [1] >>>>> https://docs.google.com/document/d/1u02Bi6sS8Fkf1s7UzRRMnvLhA477bqcyxGa0nJesqkI/edit#heading=h.75wm48pjvx0h >>>>> >>>>> >>>>> On Fri, 2 Sept 2022 at 20:17, Geoff Macartney <geoff.macart...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Peter, >>>>>> >>>>>> Thanks for such a detailed writeup of how you see this working. I fear >>>>>> I've too little experience with this sort of thing to be able to say >>>>>> anything very useful about it. My thought on the matter would be, >>>>>> let's get started with the yaml based approach and see how it goes. I >>>>>> think that experience would then give us a much better feel for what a >>>>>> really nice and usable DSL for workflows would look like (probably to >>>>>> address all the pain points of the yaml approach! :-) The outline >>>>>> above will then be a good starting point, I'm sure. >>>>>> >>>>>> Cheers >>>>>> Geoff >>>>>> >>>>>> On Thu, 1 Sept 2022 at 21:26, Peter Abramowitsch >>>>>> <pabramowit...@gmail.com> wrote: >>>>>>> >>>>>>> Hi All >>>>>>> I just wanted to clarify something in my comment the other day about >>>>>> DSLs >>>>>>> since I see that the acronym was also used in Alex's original document. >>>>>>> Unless I misunderstood, Alex was proposing to create a DSL for Brooklyn >>>>>>> using yaml as syntax and writing a code layer to translate between that >>>>>>> syntax and underlying APIs which are presumably all in Java. >>>>>>> >>>>>>> What I was suggesting was a DSL written directly in Java (I guess) >>>>>> whose >>>>>>> syntax would be that language, but whose grammar would be keywords that >>>>>>> were also Java functions. Some of these functions would be pre-defined >>>>>> in >>>>>>> the DSL, while others could be defined by the user and could use other >>>>>>> functions of the DSL. The result would be turned into a JAR file (or >>>>>>> equivalent in another platform) But during the compile phase, it >>>>>> would be >>>>>>> checked for errors, and it could be debugged line by line either >>>>>> invoking >>>>>>> live functionality or using a library of mock versions of the Brooklyn >>>>>> API. >>>>>>> >>>>>>> In this 'native' DSL one could provide different types of workflow >>>>>>> constructs as functions (In the BaseClass), taking function names as >>>>>> method >>>>>>> pointers, or using Lambdas. It would be a lot easier in Ruby or Python >>>>>>> >>>>>>> // linear >>>>>>> brooklynRun(NamedTaskMethod, NamedTaskMethod) >>>>>>> >>>>>>> // chained >>>>>>> TaskMethodA()TaskMethodB(). >>>>>>> >>>>>>> // asynchronous >>>>>>> brooklynJoin(NamedTaskMethod, NamedTaskMethod,...) >>>>>>> >>>>>>> // conditional >>>>>>> brooklynRunIf(NamedTaskMethod, NamedConditionMethod,...) >>>>>>> >>>>>>> // iterative >>>>>>> brooklynRunWhile(NamedTaskMethod, NamedConditionMethod,...) >>>>>>> brooklynRunUntil(NamedTaskMethod, NamedConditionMethod,...) >>>>>>> >>>>>>> // there could even be a utility to implement legacy syntax (this of >>>>>> course >>>>>>> would require the extra code layer I was trying to avoid) >>>>>>> runYaml(Path) >>>>>>> >>>>>>> A basic class structure might be >>>>>>> >>>>>>> // where BrooklynRecipeBase implements the utility functions including, >>>>>>> among others Join, Run, If, While, Until mentioned above >>>>>>> // and the BrooklynWorkflowInterface would dictate the functional >>>>>>> requirements for the mandatory aspects of the Recipe. >>>>>>> class MyRecipe extends BrooklynRecipeBase implements, >>>>>>> BrooklynWorkflowInterface >>>>>>> { >>>>>>> Initialize() >>>>>>> createContext() - spin up resources >>>>>>> workflow() - the main launch sequence using aspects of the DSL >>>>>>> monitoring() - an asynchronous workflow used to manage sensor output or >>>>>> for >>>>>>> whatever needs to be done while the "orchestra" is plating >>>>>>> shutdownHook() - called whenever shutdown is happening >>>>>>> } >>>>>>> >>>>>>> For those who don't like the smell of Java, the source file could just >>>>>> be >>>>>>> the contents, which would then be injected into the class framing code >>>>>>> before compilation. >>>>>>> >>>>>>> These are just ideas. I'm not familiar enough with Brooklyn in its >>>>>> current >>>>>>> implementation to be able to create realistic pseudocode. >>>>>>> >>>>>>> Peter >>>>>>> >>>>>>> On Thu, Sep 1, 2022 at 9:24 AM Geoff Macartney < >>>>>> geoff.macart...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Alex, >>>>>>>> >>>>>>>> That's great, I'll be excited to hear all about it. 7th September >>>>>>>> suits me fine; I would probably prefer 4.00 p.m. over 11.00. >>>>>>>> >>>>>>>> Cheers >>>>>>>> Geoff >>>>>>>> >>>>>>>> On Thu, 1 Sept 2022 at 12:41, Alex Heneveld <a...@cloudsoft.io> >>>>>> wrote: >>>>>>>>> >>>>>>>>> Thanks for the excellent feedback Geoff and yes there are some very >>>>>> cool >>>>>>>> and exciting things added recently -- containers, conditions, and >>>>>> terraform >>>>>>>> and kubernetes support, all of which make writing complex blueprints >>>>>> much >>>>>>>> easier. >>>>>>>>> >>>>>>>>> I'd love to host a session to showcase these. >>>>>>>>> >>>>>>>>> How does Wed 7 Sept sound? I could do 11am UK or 4pm UK -- >>>>>> depending >>>>>>>> what time suits for people who are interested. Please RSVP and >>>>>> indicate >>>>>>>> your time preference! >>>>>>>>> >>>>>>>>> Best >>>>>>>>> Alex >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, 31 Aug 2022 at 22:17, Geoff Macartney < >>>>>> geoff.macart...@gmail.com> >>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi Alex, >>>>>>>>>> >>>>>>>>>> Another thought occurred to me when reading that workflow >>>>>> proposal. You >>>>>>>> wrote >>>>>>>>>> >>>>>>>>>> "and with the recent support for container-based tasks and >>>>>> declarative >>>>>>>>>> conditions, we have taken big steps towards enabling YAML >>>>>> authorship" >>>>>>>>>> >>>>>>>>>> Unfortunately over the past while I haven't been able to keep up as >>>>>>>>>> closely as I would like with developments in Brooklyn. I'm just >>>>>>>>>> wondering if it might be possible to get together some time, on >>>>>> Google >>>>>>>>>> Meet or Zoom or whatnot, if you or a colleague could spare half an >>>>>>>>>> hour to demo some of these recent developments? But don't worry >>>>>> about >>>>>>>>>> it if you're too busy at present. >>>>>>>>>> >>>>>>>>>> Adding dev@ to this in CC for the sake of Openness. Others might >>>>>> also >>>>>>>>>> be interested! >>>>>>>>>> >>>>>>>>>> Cheers >>>>>>>>>> Geoff >>>>>>>> >>>>>> >>>>> >>
