I have merged Robin's LDAP patch. On Thu, Jan 5, 2017 at 10:39 AM, Chris Riccomini <[email protected]> wrote:
> I have found the email problem: > > https://issues.apache.org/jira/browse/AIRFLOW-734 > > Working on a fix. > > On Thu, Jan 5, 2017 at 9:10 AM, Chris Riccomini <[email protected]> > wrote: > >> Also, I'm seeing a second issue: >> >> SMTP doesn't seem to work for us anymore: >> >> [2017-01-05 15:10:13,666] {models.py:1378} ERROR - SMTP AUTH extension not >> supported by server. >> Traceback (most recent call last): >> File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1374, in >> handle_failure >> self.email_alert(error, is_retry=False) >> File "/usr/lib/python2.7/site-packages/airflow/models.py", line 1521, in >> email_alert >> send_email(task.email, title, body) >> File "/usr/lib/python2.7/site-packages/airflow/utils/email.py", line 43, >> in send_email >> return backend(to, subject, html_content, files=files, dryrun=dryrun, >> cc=cc, bcc=bcc, mime_subtype=mime_subtype) >> File "/usr/lib/python2.7/site-packages/airflow/utils/email.py", line 84, >> in send_email_smtp >> send_MIME_email(SMTP_MAIL_FROM, recipients, msg, dryrun) >> File "/usr/lib/python2.7/site-packages/airflow/utils/email.py", line 100, >> in send_MIME_email >> s.login(SMTP_USER, SMTP_PASSWORD) >> File "/usr/lib64/python2.7/smtplib.py", line 584, in login >> raise SMTPException("SMTP AUTH extension not supported by server.") >> SMTPException: SMTP AUTH extension not supported by server. >> >> This was working on 1.7.1.2. Having a look now. >> >> On Thu, Jan 5, 2017 at 9:09 AM, Chris Riccomini <[email protected]> >> wrote: >> >>> Hey Robin, >>> >>> Awesome, thanks! I love open source. :) Let me try your patch out and >>> merge it. >>> >>> Cheers, >>> Chris >>> >>> On Thu, Jan 5, 2017 at 1:34 AM, Miller, Robin < >>> [email protected]> wrote: >>> >>>> Hi Chris, >>>> >>>> >>>> I think I ran into this issue when setting up LDAP Auth in our >>>> environment (we're using very close to master as we needed some of the >>>> newer features/bugfixes). The problem turned out to be that the search was >>>> finding no results, so the line: >>>> >>>> >>>> groups_list = [regex.search(i).group(1) for i in user_groups] >>>> >>>> >>>> would fail because it had no matching groups to return. This turned out >>>> to be because Windows Active Directory (the LDAP server we're using) >>>> returned capitals "CN=" where the code expected lowercase: regex = >>>> re.compile("cn=([^,]*).*") >>>> >>>> >>>> I haven't looked it up, but Windows Active Directory is case >>>> insensitive when it comes to usernames and groups, so I wouldn't be >>>> surprised if the protocol itself is case insensitive and both "cn=" and >>>> "CN=" should be considered valid. As such I've a PR open for a simple fix >>>> to make this regex case insensitive: https://github.com/apache/incu >>>> bator-airflow/pull/1945 >>>> >>>> >>>> Hopefully this helps, >>>> >>>> Robin Miller >>>> OLIVER WYMAN >>>> [email protected]<mailto:robin.miller@a >>>> ffiliate.oliverwyman.com> >>>> www.oliverwyman.com<http://www.oliverwyman.com/> >>>> >>>> ________________________________ >>>> From: Chris Riccomini <[email protected]> >>>> Sent: 05 January 2017 00:34:27 >>>> To: [email protected] >>>> Subject: Re: Airflow 1.8.0 alpha 2 >>>> >>>> I am now running 1.8.0a2 in our dev environment. It seems to be >>>> functioning >>>> well. >>>> >>>> One issue we've hit is that the LDAP auth plugin isn't working for us >>>> anymore: >>>> >>>> Traceback (most recent call last): >>>> File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1988, in >>>> wsgi_app >>>> response = self.full_dispatch_request() >>>> File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1641, in >>>> full_dispatch_request >>>> rv = self.handle_user_exception(e) >>>> File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1544, in >>>> handle_user_exception >>>> reraise(exc_type, exc_value, tb) >>>> File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1639, in >>>> full_dispatch_request >>>> rv = self.dispatch_request() >>>> File "/usr/lib64/python2.7/site-packages/flask/app.py", line 1625, in >>>> dispatch_request >>>> return self.view_functions[rule.endpoint](**req.view_args) >>>> File "/usr/lib64/python2.7/site-packages/flask_admin/base.py", line >>>> 69, >>>> in inner >>>> return self._run_view(f, *args, **kwargs) >>>> File "/usr/lib64/python2.7/site-packages/flask_admin/base.py", line >>>> 368, >>>> in _run_view >>>> return fn(self, *args, **kwargs) >>>> File "/usr/lib/python2.7/site-packages/airflow/www/views.py", line >>>> 657, >>>> in login >>>> return airflow.login.login(self, request) >>>> File >>>> "/usr/lib/python2.7/site-packages/airflow/contrib/auth/backe >>>> nds/ldap_auth.py", >>>> line 276, in login >>>> flask_login.login_user(LdapUser(user)) >>>> File "<string>", line 4, in __init__ >>>> File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/state.py", >>>> line >>>> 306, in _initialize_instance >>>> manager.dispatch.init_failure(self, args, kwargs) >>>> File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelp >>>> ers.py", >>>> line 60, in __exit__ >>>> compat.reraise(exc_type, exc_value, exc_tb) >>>> File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/state.py", >>>> line >>>> 303, in _initialize_instance >>>> return manager.original_init(*mixed[1:], **kwargs) >>>> File >>>> "/usr/lib/python2.7/site-packages/airflow/contrib/auth/backe >>>> nds/ldap_auth.py", >>>> line 148, in __init__ >>>> user.username) >>>> File >>>> "/usr/lib/python2.7/site-packages/airflow/contrib/auth/backe >>>> nds/ldap_auth.py", >>>> line 106, in groups_user >>>> groups_list = [regex.search(i).group(1) for i in user_groups] >>>> AttributeError: 'NoneType' object has no attribute 'group' >>>> >>>> I believe it's from this patch: >>>> >>>> https://github.com/apache/incubator-airflow/commit/d6d3f5367 >>>> 3ba3736d7a858531823933cfef2bb4e >>>> >>>> I haven't dug into it yet. We'll need to fix it, though. In the >>>> meantime, I >>>> just commented out the call to `groups_user`. More to come. >>>> >>>> On Wed, Jan 4, 2017 at 12:32 PM, Alex Van Boxel <[email protected]> >>>> wrote: >>>> >>>> > I have another fix that certainly need to be in the final release, >>>> but not >>>> > ready to merge due to failed tests: >>>> > >>>> > https://github.com/apache/incubator-airflow/pull/1961 >>>> > >>>> > >>>> > >>>> > >>>> > On Wed, Jan 4, 2017 at 8:48 PM Chris Riccomini <[email protected] >>>> > >>>> > wrote: >>>> > >>>> > > Great, I will work toward deploying this in our dev cluster today. >>>> :D >>>> > > >>>> > > On Wed, Jan 4, 2017 at 11:47 AM, Bolke de Bruin <[email protected]> >>>> > wrote: >>>> > > >>>> > > > Some issues remain: >>>> > > > >>>> > > > * one_failed not executed as dag run is marked failed seemingly >>>> > > > prematurely (@chris yes you should see this, see below for an >>>> example >>>> > > that >>>> > > > is not working properly), confirmed regression >>>> > > > * celery instability Alex >>>> > > > * Wrong DAG state after failure inside branch >>>> > > > >>>> > > > So Alpha 2 is definitely not ready for production, but please do >>>> put in >>>> > > > your canary dags and let them run. I am still quite concerned >>>> about the >>>> > > > scheduler integrity and stability. >>>> > > > >>>> > > > - Bolke >>>> > > > >>>> > > > >>>> > > > >>>> > > > one_failed_not_executed.py >>>> > > > ==== >>>> > > > from airflow import DAG >>>> > > > from airflow.operators.bash_operator import BashOperator >>>> > > > from airflow.operators.dummy_operator import DummyOperator >>>> > > > from datetime import datetime, timedelta >>>> > > > >>>> > > > default_args = { >>>> > > > 'owner': 'airflow', >>>> > > > 'depends_on_past': False, >>>> > > > 'start_date': datetime(2016,10,5,19), >>>> > > > 'email': ['[email protected]'], >>>> > > > 'email_on_failure': False, >>>> > > > 'email_on_retry': False, >>>> > > > 'retries': 1, >>>> > > > 'retry_delay': timedelta(seconds=1), >>>> > > > } >>>> > > > >>>> > > > dag = DAG('tutorial', default_args=default_args, >>>> > > schedule_interval='@once') >>>> > > > >>>> > > > task1 = BashOperator( >>>> > > > task_id='first_one', >>>> > > > bash_command='date', >>>> > > > dag=dag) >>>> > > > >>>> > > > task2 = BashOperator( >>>> > > > task_id='second_one', >>>> > > > bash_command='this_should_not_work', >>>> > > > dag=dag) >>>> > > > >>>> > > > task2.set_upstream(task1) >>>> > > > >>>> > > > >>>> > > > >>>> > > > task3 = BashOperator( >>>> > > > task_id='third_one', >>>> > > > bash_command='random_command_third', >>>> > > > dag=dag) >>>> > > > >>>> > > > task3.set_upstream(task2) >>>> > > > >>>> > > > fail_task = DummyOperator( >>>> > > > task_id='one_failed', >>>> > > > trigger_rule='one_failed', >>>> > > > dag=dag) >>>> > > > >>>> > > > fail_task.set_upstream([task1,task2,task3]) >>>> > > > >>>> > > > > On 4 Jan 2017, at 20:35, Chris Riccomini <[email protected] >>>> > >>>> > > wrote: >>>> > > > > >>>> > > > > Bolke, can you describe the current state of the alpha 2 >>>> release? I >>>> > saw >>>> > > > > some comments from Alex yesterday about celery instability. If >>>> I'm >>>> > > > running >>>> > > > > on LocalExecutor, should I be seeing any issues? >>>> > > > > >>>> > > > > On Wed, Jan 4, 2017 at 8:20 AM, Bolke de Bruin < >>>> [email protected]> >>>> > > > wrote: >>>> > > > > >>>> > > > >> Hi All, >>>> > > > >> >>>> > > > >> I have put up Airflow 1.8.0 alpha 2 in >>>> https://people.apache.org/~ >>>> > > > bolke/ < >>>> > > > >> https://people.apache.org/~bolke/> . >>>> > > > >> >>>> > > > >> Note: This still cannot be considered an Apache release. >>>> Working on >>>> > > > this. >>>> > > > >> >>>> > > > >> This build is signed (note it is served over https). >>>> > > > >> >>>> > > > >> Changes are in the area of scheduler stability. >>>> > > > >> >>>> > > > >> - Bolke >>>> > > > >>>> > > > >>>> > > >>>> > -- >>>> > _/ >>>> > _/ Alex Van Boxel >>>> > >>>> >>>> ________________________________ >>>> This e-mail and any attachments may be confidential or legally >>>> privileged. If you received this message in error or are not the intended >>>> recipient, you should destroy the e-mail message and any attachments or >>>> copies, and you are prohibited from retaining, distributing, disclosing or >>>> using any information contained herein. Please inform us of the erroneous >>>> delivery by return e-mail. Thank you for your cooperation. >>>> >>> >>> >> >
