My intuition is that is comes from some Postgres or less likely but possible environmental setting (os, network, ...) on your end. Clearly the stack trace you share says that the server closed the connection during the cursor's execute method. I bet you that you can reproduce with a vanilla query using the same python driver outside of Airflow, and most likely with any other API in other languages.
Good luck, Max On Fri, Sep 2, 2016 at 12:58 AM, Arthur Purvis <[email protected]> wrote: > Hi - > > I am experiencing the following situation while evaluating airflow; > wondering if this rings any bells with anyone: > > > 1. *airflow 1.7.1.2* is running in a *docker* container on amazon *ECS* > 2. *dagrun_timeout* = timedelta(hours=24) for this DAG > 3. running redshift queries with the PostGresOperator works fine > > *except when that query takes over two hours*, and then the query is *cut > off* *exactly at the two hour mark* with this message > > [2016-09-02 00:32:14,307] {models.py:1286} ERROR - server closed the > connection unexpectedly > This probably means the server terminated abnormally > before or while processing the request. > Traceback (most recent call last): > File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line > 1245, in run > result = task_copy.execute(context=context) > File > "/usr/local/lib/python2.7/dist-packages/airflow/ > operators/postgres_operator.py", > line 39, in execute > self.hook.run(self.sql, self.autocommit, parameters=self.parameters) > File > "/usr/local/lib/python2.7/dist-packages/airflow/hooks/dbapi_hook.py", line > 132, in run > cur.execute(s) > OperationalError: server closed the connection unexpectedly > This probably means the server terminated abnormally > before or while processing the request. > > > > We are 100% sure that this account is able to run long queries in redshift > because we run these same queries from outside airflow from a regular AWS > EC2 instance (not a docker container) without issue. > > The *exactly *two hours says to me that there is some setting somewhere - > either in airflow itself or in the postgresoperator or in the docker TCP > settings (or somewhere else) - that is going awry, but I cannot sort out > what it could be. It doesn't seem to be an issue with celery, because > other jobs are running that polling for 6+ hours. > > If this rings any bells for anyone I would love to hear about it. > > Thanks for reading. >
