I've run into a regression with the webserver. It looks like the --pid argument is no longer honored in 1.8.1. The pid file is not being written out! As a result, monitd, which watches the processes mentioned in the pid file, keep trying to spawn webservers.
HISTTIMEFORMAT="%d/%m/%y %T " PYTHONPATH=/usr/local/agari/ep-pipeline/production/current/analysis/cluster/:/usr/local/agari/ep-pipeline/production/current/analysis/lookups/ TMP=/data/tmp AIRFLOW_HOME=/data/airflow PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin airflow webserver -p 8080 --pid /data/airflow/pids/airflow-webserver.pid The "upgrade" process for 1.8.1. is not simply "pip install 1.8.1. tarball". It requires a "pip uninstall" of the previous 1.8.0 version followed by an new installation of 1.8.1. This could have pulled in some new dependencies that broke how this works. On Tue, Apr 18, 2017 at 12:32 PM, siddharth anand <[email protected]> wrote: > Hmn.. it always worked for me for any of the releases we installed. I > install `pip install <link to release tarball>` > > -s > > On Tue, Apr 18, 2017 at 10:44 AM, Chris Riccomini <[email protected]> > wrote: > >> @Sid, how do you enable the versioning? I've never been able to get this >> to >> work in my environment. It always shows "Not available", even with 1.8.0. >> >> On Mon, Apr 17, 2017 at 11:18 PM, Bolke de Bruin <[email protected]> >> wrote: >> >> > Hey Alex, >> > >> > I agree with you that they are nice to have, but as you mentioned they >> are >> > not blockers. As we are moving towards time based releases I suggest >> > marking them for 1.8.2 and cherry-picking them in your production. >> > >> > - Bolke. >> > >> > > On 18 Apr 2017, at 00:02, Alex Guziel <[email protected] >> D> >> > wrote: >> > > >> > > Sorry about that. FWIW, these were recent and I don't think they were >> > > blockers but are nice to fix. Particularly, the tree one was forgotten >> > > about. I remember seeing it at the Airflow hackathon but I guess I >> forgot >> > > to correct it. >> > > >> > > On Mon, Apr 17, 2017 at 12:17 PM, Chris Riccomini < >> [email protected] >> > > >> > > wrote: >> > > >> > >> :(:(:( Why was this not included in 1.8.1 JIRA? I've been emailing >> the >> > list >> > >> all last week.... >> > >> >> > >> On Mon, Apr 17, 2017 at 11:28 AM, Alex Guziel < >> > >> [email protected]> wrote: >> > >> >> > >>> I would say to include [1074] ( >> > >>> https://github.com/apache/incubator-airflow/pull/2221) so we don't >> > have >> > >> a >> > >>> regression in the release after. I would also say >> > >>> https://github.com/apache/incubator-airflow/pull/2241 is semi >> > important >> > >>> but >> > >>> less so. >> > >>> >> > >>> On Mon, Apr 17, 2017 at 11:24 AM, Chris Riccomini < >> > [email protected] >> > >>> >> > >>> wrote: >> > >>> >> > >>>> Dear All, >> > >>>> >> > >>>> I have been able to make the Airflow 1.8.1 RC0 available at: >> > >>>> https://dist.apache.org/repos/dist/dev/incubator/airflow, public >> keys >> > >>> are >> > >>>> available at https://dist.apache.org/repos/ >> > >>> dist/release/incubator/airflow. >> > >>>> >> > >>>> Issues fixed: >> > >>>> >> > >>>> [AIRFLOW-1062] DagRun#find returns wrong result if external_trigg >> > >>>> [AIRFLOW-1054] Fix broken import on test_dag >> > >>>> [AIRFLOW-1050] Retries ignored - regression >> > >>>> [AIRFLOW-1033] TypeError: can't compare datetime.datetime to None >> > >>>> [AIRFLOW-1030] HttpHook error when creating HttpSensor >> > >>>> [AIRFLOW-1017] get_task_instance should return None instead of th >> > >>>> [AIRFLOW-1011] Fix bug in BackfillJob._execute() for SubDAGs >> > >>>> [AIRFLOW-1001] Landing Time shows "unsupported operand type(s) fo >> > >>>> [AIRFLOW-1000] Rebrand to Apache Airflow instead of Airflow >> > >>>> [AIRFLOW-989] Clear Task Regression >> > >>>> [AIRFLOW-974] airflow.util.file mkdir has a race condition >> > >>>> [AIRFLOW-906] Update Code icon from lightning bolt to file >> > >>>> [AIRFLOW-858] Configurable database name for DB operators >> > >>>> [AIRFLOW-853] ssh_execute_operator.py stdout decode default to A >> > >>>> [AIRFLOW-832] Fix debug server >> > >>>> [AIRFLOW-817] Trigger dag fails when using CLI + API >> > >>>> [AIRFLOW-816] Make sure to pull nvd3 from local resources >> > >>>> [AIRFLOW-815] Add previous/next execution dates to available def >> > >>>> [AIRFLOW-813] Fix unterminated unit tests in tests.job (tests/jo >> > >>>> [AIRFLOW-812] Scheduler job terminates when there is no dag file >> > >>>> [AIRFLOW-806] UI should properly ignore DAG doc when it is None >> > >>>> [AIRFLOW-794] Consistent access to DAGS_FOLDER and SQL_ALCHEMY_C >> > >>>> [AIRFLOW-785] ImportError if cgroupspy is not installed >> > >>>> [AIRFLOW-784] Cannot install with funcsigs > 1.0.0 >> > >>>> [AIRFLOW-780] The UI no longer shows broken DAGs >> > >>>> [AIRFLOW-777] dag_is_running is initlialized to True instead of >> > >>>> [AIRFLOW-719] Skipped operations make DAG finish prematurely >> > >>>> [AIRFLOW-694] Empty env vars do not overwrite non-empty config v >> > >>>> [AIRFLOW-139] Executing VACUUM with PostgresOperator >> > >>>> [AIRFLOW-111] DAG concurrency is not honored >> > >>>> [AIRFLOW-88] Improve clarity Travis CI reports >> > >>>> >> > >>>> I would like to raise a VOTE for releasing 1.8.1 based on release >> > >>> candidate >> > >>>> 0, i.e. just renaming release candidate 0 to 1.8.1 release. >> > >>>> >> > >>>> Please respond to this email by: >> > >>>> >> > >>>> +1,0,-1 with *binding* if you are a PMC member or *non-binding* if >> you >> > >>> are >> > >>>> not. >> > >>>> >> > >>>> Vote will run for 72 hours (ends this Thursday). >> > >>>> >> > >>>> Thanks! >> > >>>> Chris >> > >>>> >> > >>>> My VOTE: +1 (binding) >> > >>>> >> > >>> >> > >> >> > >> > >> > >
