fixed here: https://github.com/apache/incubator-airflow/pull/2513
On Mon, Sep 25, 2017 at 10:28 AM, Driesprong, Fokko <[email protected]> wrote: > Hi Shoumitra, > > So if I understand correctly, the query finishes but the connection does > not close. What is the result of the query? Maybe you can also share some > code of how you use the PostgresOperator the communicate with Redshift. > > Regarding your config. The sequential executor is only recommended when > debugging, since it runs in process and only allows you to execute one task > at the time. I would recommend upgrading to the LocalExecutor and use an > AWS managed Postgres for the database. > > Let me know. > > Cheers, Fokko > > 2017-09-25 7:44 GMT+02:00 Shoumitra Srivastava <[email protected]>: > > > Hey guys, > > > > I am currently using Airflow 1.8.2 to schedule some EMR tasks and then > > execute some long running queries on our Redshift cluster. For that > purpose > > I am using the postgres_operator. The queries take about 30 minutes to > run. > > However, once they are done, the connection never closes and the operator > > runs for an hour and a half more till its terminated at the 2 hour mark > > every time. The logs say that the server closed the connection > > unexpectedly. I've checked the logs on Redshift's end and it shows the > > queries have run and the connection has been closed. Somehow, that is > never > > communicated back to Airflow. Any directions on what more I could check? > To > > give some more info, my Airflow installation is an extension of the > > https://github.com/puckel/docker-airflow docker image, is run in an ECS > > cluster and has SQLite as backend since I am still testing Airflow out. I > > am also using Sequential Executor right now. I would appreciate any help > in > > this matter. > > > > thanks, > > Shoumitra > > >
