We are using Composer (Airflow) to schedule and run the Dataflow jobs... Using the Python SDK, with small changes no the Composer (Airflow) DataFlow PythonOperator, to force it to use Python 3... It is working fine and creating a new Dataflow job every 30 minutes...
On Mon, Apr 6, 2020 at 10:33 PM Marco Mistroni <[email protected]> wrote: > Right.. tx Andre. So presumably the flow of action will b > - create dflow template > -create CLF that invokes it > - create cold scheduler job that invokes function? > > Kind regards > > On Mon, Apr 6, 2020, 2:14 PM André Rocha Silva < > [email protected]> wrote: > >> Marco >> >> If you are already using GCP, I suggest you use the cloud scheduler. It >> is like a cron job completely serverless. >> >> If you need some extra help, let me know. >> >> On Mon, Apr 6, 2020 at 4:38 AM deepak kumar <[email protected]> wrote: >> >>> We have used composer (airlfow) successfully to schedule Dataflow jobs. >>> Please let me know if you would need details around it. >>> >>> Thanks >>> Deepak >>> >>> On Sun, Apr 5, 2020 at 7:56 PM Joshua B. Harrison < >>> [email protected]> wrote: >>> >>>> Hi Marco, >>>> >>>> I've ended using a VM running Luigi to schedule jobs. I use the data >>>> flow Python API to execute stored templates. >>>> >>>> I can give you more details if you’re interested. >>>> >>>> Best, >>>> Joshua >>>> >>>> On Sun, Apr 5, 2020 at 5:02 AM Marco Mistroni <[email protected]> >>>> wrote: >>>> >>>>> HI all >>>>> sorry for this partially OT but has anyone been successful in >>>>> scheduling dataflow job on GCP? >>>>> I have tried the CloudFunction approach (following few eamples on the >>>>> web) but it didnt work out for me - the cloud function keep on giving me >>>>> an >>>>> INVALID ARGUMENT - which i could not debug >>>>> >>>>> So i was wondering if anyone has been successful and can provide me >>>>> an example >>>>> >>>>> kind regards >>>>> Marco >>>>> >>>>> -- >>>> Joshua Harrison | Software Engineer | [email protected] >>>> <[email protected]> | 404-433-0242 >>>> >>> >> >> -- >> >> *ANDRÉ ROCHA SILVA* >> * DATA ENGINEER* >> (48) 3181-0611 >> >> <https://www.linkedin.com/in/andre-rocha-silva/> /andre-rocha-silva/ >> <http://portaltelemedicina.com.br/> >> <https://www.youtube.com/channel/UC0KH36-OXHFIKjlRY2GyAtQ> >> <https://pt-br.facebook.com/PortalTelemedicina/> >> <https://www.linkedin.com/company/9426084/> >> >> -- Soliman ElSaber Data Engineer www.mindvalley.com
