Hi guys, Thank you for your quick replies. Seems like Arthur's pull request addresses exactly what I am facing. Hopefully, it gets merged and I can move ahead.
thanks, Shoumitra On Mon, Sep 25, 2017 at 8:25 AM, Arthur Purvis <[email protected]> wrote: > fixed here: https://github.com/apache/incubator-airflow/pull/2513 > > On Mon, Sep 25, 2017 at 10:28 AM, Driesprong, Fokko <[email protected]> > wrote: > >> Hi Shoumitra, >> >> So if I understand correctly, the query finishes but the connection does >> not close. What is the result of the query? Maybe you can also share some >> code of how you use the PostgresOperator the communicate with Redshift. >> >> Regarding your config. The sequential executor is only recommended when >> debugging, since it runs in process and only allows you to execute one >> task >> at the time. I would recommend upgrading to the LocalExecutor and use an >> AWS managed Postgres for the database. >> >> Let me know. >> >> Cheers, Fokko >> >> 2017-09-25 7:44 GMT+02:00 Shoumitra Srivastava <[email protected]>: >> >> > Hey guys, >> > >> > I am currently using Airflow 1.8.2 to schedule some EMR tasks and then >> > execute some long running queries on our Redshift cluster. For that >> purpose >> > I am using the postgres_operator. The queries take about 30 minutes to >> run. >> > However, once they are done, the connection never closes and the >> operator >> > runs for an hour and a half more till its terminated at the 2 hour mark >> > every time. The logs say that the server closed the connection >> > unexpectedly. I've checked the logs on Redshift's end and it shows the >> > queries have run and the connection has been closed. Somehow, that is >> never >> > communicated back to Airflow. Any directions on what more I could >> check? To >> > give some more info, my Airflow installation is an extension of the >> > https://github.com/puckel/docker-airflow docker image, is run in an ECS >> > cluster and has SQLite as backend since I am still testing Airflow out. >> I >> > am also using Sequential Executor right now. I would appreciate any >> help in >> > this matter. >> > >> > thanks, >> > Shoumitra >> > >> > >
