Thanks will give it a go On Mon, Apr 6, 2020, 3:39 PM Soliman ElSaber <[email protected]> wrote:
> We are using Composer (Airflow) to schedule and run the Dataflow jobs... > Using the Python SDK, with small changes no the Composer (Airflow) > DataFlowPythonOperator, to force it to use Python 3... > It is working fine and creating a new Dataflow job every 30 minutes... > > On Mon, Apr 6, 2020 at 10:33 PM Marco Mistroni <[email protected]> > wrote: > >> Right.. tx Andre. So presumably the flow of action will b >> - create dflow template >> -create CLF that invokes it >> - create cold scheduler job that invokes function? >> >> Kind regards >> >> On Mon, Apr 6, 2020, 2:14 PM André Rocha Silva < >> [email protected]> wrote: >> >>> Marco >>> >>> If you are already using GCP, I suggest you use the cloud scheduler. It >>> is like a cron job completely serverless. >>> >>> If you need some extra help, let me know. >>> >>> On Mon, Apr 6, 2020 at 4:38 AM deepak kumar <[email protected]> wrote: >>> >>>> We have used composer (airlfow) successfully to schedule Dataflow jobs. >>>> Please let me know if you would need details around it. >>>> >>>> Thanks >>>> Deepak >>>> >>>> On Sun, Apr 5, 2020 at 7:56 PM Joshua B. Harrison < >>>> [email protected]> wrote: >>>> >>>>> Hi Marco, >>>>> >>>>> I've ended using a VM running Luigi to schedule jobs. I use the data >>>>> flow Python API to execute stored templates. >>>>> >>>>> I can give you more details if you’re interested. >>>>> >>>>> Best, >>>>> Joshua >>>>> >>>>> On Sun, Apr 5, 2020 at 5:02 AM Marco Mistroni <[email protected]> >>>>> wrote: >>>>> >>>>>> HI all >>>>>> sorry for this partially OT but has anyone been successful in >>>>>> scheduling dataflow job on GCP? >>>>>> I have tried the CloudFunction approach (following few eamples on the >>>>>> web) but it didnt work out for me - the cloud function keep on giving me >>>>>> an >>>>>> INVALID ARGUMENT - which i could not debug >>>>>> >>>>>> So i was wondering if anyone has been successful and can provide me >>>>>> an example >>>>>> >>>>>> kind regards >>>>>> Marco >>>>>> >>>>>> -- >>>>> Joshua Harrison | Software Engineer | [email protected] >>>>> <[email protected]> | 404-433-0242 >>>>> >>>> >>> >>> -- >>> >>> *ANDRÉ ROCHA SILVA* >>> * DATA ENGINEER* >>> (48) 3181-0611 >>> >>> <https://www.linkedin.com/in/andre-rocha-silva/> /andre-rocha-silva/ >>> <http://portaltelemedicina.com.br/> >>> <https://www.youtube.com/channel/UC0KH36-OXHFIKjlRY2GyAtQ> >>> <https://pt-br.facebook.com/PortalTelemedicina/> >>> <https://www.linkedin.com/company/9426084/> >>> >>> > > -- > Soliman ElSaber > Data Engineer > www.mindvalley.com >
