Flask-AppBuilder has pinned versions of Click & Flask-Login in 1.10.0

[Kyle Hamlin]

Hi,

Today I was trying to upgrade Airflow to 1.10.0 and it appears that there
are some version conflicts with click and flask-login. I uncovered these
because I use Pipenv to manage our project's dependencies. You can see here
that Flask-AppBuilder pins click==6.7 and flask-login>=0.3,<0.5

https://github.com/dpgaspar/Flask-AppBuilder/blob/master/setup.py#L37-L47

I'm able to force pipenv to install click==6.7 because that is not pinned
in Airflow's setup.py, but I can do nothing about flask-login because
Airflow pins the flask-login version:
https://github.com/apache/incubator-airflow/blob/master/setup.py#L304

This prevents me from being able to upgrade to 1.10.0.

*Pipenv's Graphed project dependencies (conflicts highlighted):*

apache-airflow==1.10.0
  - alembic [required: >=0.8.3,<0.9, installed: 0.8.10]
- Mako [required: Any, installed: 1.0.7]
  - MarkupSafe [required: >=0.9.2, installed: 1.0]
- python-editor [required: >=0.3, installed: 1.0.3]
- SQLAlchemy [required: >=0.7.6, installed: 1.2.12]
  - bleach [required: ==2.1.2, installed: 2.1.2]
- html5lib [required:
>=0.pre,!=1.0b8,!=1.0b7,!=1.0b6,!=1.0b5,!=1.0b4,!=1.0b3,!=1.0b2,!=1.0b1,
installed: 1.0.1]
  - six [required: >=1.9, installed: 1.11.0]
  - webencodings [required: Any, installed: 0.5.1]
- six [required: Any, installed: 1.11.0]
  - configparser [required: >=3.5.0,<3.6.0, installed: 3.5.0]
  - croniter [required: >=0.3.17,<0.4, installed: 0.3.25]
- python-dateutil [required: Any, installed: 2.7.3]
  - six [required: >=1.5, installed: 1.11.0]
  - dill [required: >=0.2.2,<0.3, installed: 0.2.8.2]
  - flask [required: >=0.12.4,<0.13, installed: 0.12.4]
- click [required: >=2.0, installed: 7.0]
- itsdangerous [required: >=0.21, installed: 0.24]
- Jinja2 [required: >=2.4, installed: 2.8.1]
  - MarkupSafe [required: Any, installed: 1.0]
- Werkzeug [required: >=0.7, installed: 0.14.1]
  - flask-admin [required: ==1.4.1, installed: 1.4.1]
- Flask [required: >=0.7, installed: 0.12.4]
  - click [required: >=2.0, installed: 7.0]
  - itsdangerous [required: >=0.21, installed: 0.24]
  - Jinja2 [required: >=2.4, installed: 2.8.1]
- MarkupSafe [required: Any, installed: 1.0]
  - Werkzeug [required: >=0.7, installed: 0.14.1]
- wtforms [required: Any, installed: 2.2.1]
  - flask-appbuilder [required: >=1.11.1,<2.0.0, installed: 1.12.0]
- click [required: ==6.7, installed: 7.0]
- colorama [required: ==0.3.9, installed: 0.3.9]
- Flask [required: >=0.10.0,<0.12.99, installed: 0.12.4]
  - click [required: >=2.0, installed: 7.0]
  - itsdangerous [required: >=0.21, installed: 0.24]
  - Jinja2 [required: >=2.4, installed: 2.8.1]
- MarkupSafe [required: Any, installed: 1.0]
  - Werkzeug [required: >=0.7, installed: 0.14.1]
- Flask-Babel [required: ==0.11.1, installed: 0.11.1]
  - Babel [required: >=2.3, installed: 2.6.0]
- pytz [required: >=0a, installed: 2018.5]
  - Flask [required: Any, installed: 0.12.4]
- click [required: >=2.0, installed: 7.0]
- itsdangerous [required: >=0.21, installed: 0.24]
- Jinja2 [required: >=2.4, installed: 2.8.1]
  - MarkupSafe [required: Any, installed: 1.0]
- Werkzeug [required: >=0.7, installed: 0.14.1]
  - Jinja2 [required: >=2.5, installed: 2.8.1]
- MarkupSafe [required: Any, installed: 1.0]
- Flask-Login [required: >=0.3,<0.5, installed: 0.2.11]
  - Flask [required: Any, installed: 0.12.4]
- click [required: >=2.0, installed: 7.0]
- itsdangerous [required: >=0.21, installed: 0.24]
- Jinja2 [required: >=2.4, installed: 2.8.1]
  - MarkupSafe [required: Any, installed: 1.0]
- Werkzeug [required: >=0.7, installed: 0.14.1]
- Flask-OpenID [required: ==1.2.5, installed: 1.2.5]
  - Flask [required: >=0.10.1, installed: 0.12.4]
- click [required: >=2.0, installed: 7.0]
- itsdangerous [required: >=0.21, installed: 0.24]
- Jinja2 [required: >=2.4, installed: 2.8.1]
  - MarkupSafe [required: Any, installed: 1.0]
- Werkzeug [required: >=0.7, installed: 0.14.1]
  - python3-openid [required: >=2.0, installed: 3.1.0]
- defusedxml [required: Any, installed: 0.5.0]
- Flask-SQLAlchemy [required: ==2.1, installed: 2.1]
  - Flask [required: >=0.10, installed: 0.12.4]
- click [required: >=2.0, installed: 7.0]
- itsdangerous [required: >=0.21, installed: 0.24]
- Jinja2 [required: >=2.4, installed: 2.8.1]
  - MarkupSafe [required: Any, installed: 1.0]
- Werkzeug [required: >=0.7, installed: 0.14.1]
  - SQLAlchemy [required: >=0.7, installed: 1.2.12]
- Flask-WTF [required: ==0.14.2, installed: 0.14.2]
  - Flask [required: Any, installed: 0.12.4]
- click [required: >=2.0, installed: 7.0]
- itsdangerous [required: >=0.21, installed: 0.24]
- Jinja2 [required: 

Re: "setup.py test" is being naughty

[Jarek Potiuk]

Local testing works well for a number of unit tests when run from the IDE.
We of course run full suite of tests via docker environment but our own
test classess/modules are run using local python environment. It's the
easiest way to configure local python virtualenv with IntelliJ/Pycharm for
one. You can - in recent version of PyCharm/IntelliJ - have docker python
environment setup, but there are certain downsides of using it
(speed/mounting local volumes with sources etc.).

So I think we should not really discourage running at least some tests
locally. Maybe (if there are not many of those) we could identify the tests
which require the full-blown docker environment and mark them with
skipUnless and only have them executed when we are inside dockerized
environment for unit tests ?

J.


On Wed, Oct 3, 2018 at 1:48 PM Holden Karau  wrote:

> I think (in the short term) discontinuing local testing and telling folks
> to use the docker based approach makes more sense (many of the tests have a
> complex set of dependencies that don't make sense to try and test locally).
> What do other folks think?
>
> On Wed, Oct 3, 2018 at 4:45 AM EKC (Erik Cederstrand)
>  wrote:
>
> > The test suite is also trying to create /usr/local/bin/airflow, which
> > means I can't run the test suite on a machine that actually uses
> > /usr/local/bin/airflow. And the default config file doesn't find the
> MySQL
> > server I set up locally. I'm trying the Docker-based test environment
> now.
> >
> >
> > It seems the local test setup either needs polishing or should be
> > discontinued.
> >
> >
> > Erik
> >
> > 
> > From: EKC (Erik Cederstrand)
> > Sent: Wednesday, October 3, 2018 12:01:00 PM
> > To: dev@airflow.incubator.apache.org
> > Subject: "setup.py test" is being naughty
> >
> >
> > Hi all,
> >
> >
> > I wanted to contribute a simple patch, and as a good open source citizen
> I
> > wanted to also contribute a test. So I git clone from GitHub, create a
> > virtualenv and run "setup.py test". First experience is that my
> > /etc/krb5.conf is overwritten, which means my account is locked out of
> all
> > systems here at work. I recovered from that, only to find out that
> > ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub were also overwritten. Now I'm not
> very
> > amused.
> >
> >
> > Did I miss something in CONTRIBUTING.md?
> >
> >
> > Erik
> >
>
>
> --
> Twitter: https://twitter.com/holdenkarau
> Books (Learning Spark, High Performance Spark, etc.):
> https://amzn.to/2MaRAG9  
> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>


-- 

*Jarek Potiuk, Principal Software Engineer*
Mobile: +48 660 796 129

Re: "setup.py test" is being naughty

[Holden Karau]

I think (in the short term) discontinuing local testing and telling folks
to use the docker based approach makes more sense (many of the tests have a
complex set of dependencies that don't make sense to try and test locally).
What do other folks think?

On Wed, Oct 3, 2018 at 4:45 AM EKC (Erik Cederstrand)
 wrote:

> The test suite is also trying to create /usr/local/bin/airflow, which
> means I can't run the test suite on a machine that actually uses
> /usr/local/bin/airflow. And the default config file doesn't find the MySQL
> server I set up locally. I'm trying the Docker-based test environment now.
>
>
> It seems the local test setup either needs polishing or should be
> discontinued.
>
>
> Erik
>
> 
> From: EKC (Erik Cederstrand)
> Sent: Wednesday, October 3, 2018 12:01:00 PM
> To: dev@airflow.incubator.apache.org
> Subject: "setup.py test" is being naughty
>
>
> Hi all,
>
>
> I wanted to contribute a simple patch, and as a good open source citizen I
> wanted to also contribute a test. So I git clone from GitHub, create a
> virtualenv and run "setup.py test". First experience is that my
> /etc/krb5.conf is overwritten, which means my account is locked out of all
> systems here at work. I recovered from that, only to find out that
> ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub were also overwritten. Now I'm not very
> amused.
>
>
> Did I miss something in CONTRIBUTING.md?
>
>
> Erik
>


-- 
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Re: "setup.py test" is being naughty

[EKC (Erik Cederstrand)]

The test suite is also trying to create /usr/local/bin/airflow, which means I 
can't run the test suite on a machine that actually uses 
/usr/local/bin/airflow. And the default config file doesn't find the MySQL 
server I set up locally. I'm trying the Docker-based test environment now.


It seems the local test setup either needs polishing or should be discontinued.


Erik


From: EKC (Erik Cederstrand)
Sent: Wednesday, October 3, 2018 12:01:00 PM
To: dev@airflow.incubator.apache.org
Subject: "setup.py test" is being naughty


Hi all,


I wanted to contribute a simple patch, and as a good open source citizen I 
wanted to also contribute a test. So I git clone from GitHub, create a 
virtualenv and run "setup.py test". First experience is that my /etc/krb5.conf 
is overwritten, which means my account is locked out of all systems here at 
work. I recovered from that, only to find out that ~/.ssh/id_rsa and 
~/.ssh/id_rsa.pub were also overwritten. Now I'm not very amused.


Did I miss something in CONTRIBUTING.md?


Erik

Re: "setup.py test" is being naughty

[Holden Karau]

That's a really good point. The docs suggest running in docker but we don't
warn people about the downsides of running the tests outside of docker and
we should probably at least warn folks if it looks like they are running on
bare metal.

On Wed, Oct 3, 2018 at 3:01 AM EKC (Erik Cederstrand)
 wrote:

> Hi all,
>
>
> I wanted to contribute a simple patch, and as a good open source citizen I
> wanted to also contribute a test. So I git clone from GitHub, create a
> virtualenv and run "setup.py test". First experience is that my
> /etc/krb5.conf is overwritten, which means my account is locked out of all
> systems here at work. I recovered from that, only to find out that
> ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub were also overwritten. Now I'm not very
> amused.
>
>
> Did I miss something in CONTRIBUTING.md?
>
>
> Erik
>


-- 
Twitter: https://twitter.com/holdenkarau
Books (Learning Spark, High Performance Spark, etc.):
https://amzn.to/2MaRAG9  
YouTube Live Streams: https://www.youtube.com/user/holdenkarau

Airflow Docs - RTD vs Apache Site

[Kaxil Naik]

Hi all,

Continuing discussion from Slack, many users have had the problem with
looking at a wrong version of the documentation. Currently, our docs on
apache.airflow.org don't properly state version. Although we have specified
this info on our Github readme and confluence, there has still been lots of
confusion among the new users who try to google for the docs and are
pointed to airflow.apache.org site which doesn't have version info.

The problem currently with a.a.o site is it needs to be manually built and
only has stable version docs. We can do 2 things if we don't want to
redirect a.a.o with RTD: (1) Maintain History on our static a.a.o site (2)
Point a.a.o site to RTD docs, so a.a.o would point to RTD docs i.e. add the
domain to RTD site

Ash has also suggested another approach:

> Apache Infra run a jenkins instance (or other build bot type things) that
> we might be able to use for autobuilding docs if we want?



Let's discuss this and decide on a single-approach that is user-friendly.

NB: I will be busy for a month, hence won't be able to actively help with
this, so please feel free to contribute/commit after an approach is
finalized.

Regards,
Kaxil

"setup.py test" is being naughty

[EKC (Erik Cederstrand)]

Hi all,


I wanted to contribute a simple patch, and as a good open source citizen I 
wanted to also contribute a test. So I git clone from GitHub, create a 
virtualenv and run "setup.py test". First experience is that my /etc/krb5.conf 
is overwritten, which means my account is locked out of all systems here at 
work. I recovered from that, only to find out that ~/.ssh/id_rsa and 
~/.ssh/id_rsa.pub were also overwritten. Now I'm not very amused.


Did I miss something in CONTRIBUTING.md?


Erik

Re: Task is Stuck in Up_For_Retry

[raman gupta]

Hi All,

On furter investigation we found that this issue is reproduced if task
retry_delay is set to very low number say 10 seconds.
 All the retries of a taks get the same key by executor (LocalExecutor )
which is composed of dagid, taskid and execution date. So we are hitting
the scenario where scheduler schedules the next task run while the
executor's event of previous run is not yet processed. So in jobs.py's
process_executor_events function, scheduler associates the last run's
executor event with the current run(which might be in the queued state )
and marks it as failed/up_for_retry.
Due to this task state's follow
up_for_retry->scheduled->queued->failed/up_for_retry transition instead of
following   up_for_retry->scheduled->queued->running->failed/up_for_retry.
And we get to see the logs like "Task is not being able to run" and
"executor reports the task as finished but it is still in queued state"
We have raised a JIRA for this
https://issues.apache.org/jira/browse/AIRFLOW-3136 and are also looking
into potential fix.

Thanks,
Raman Gupta

On Fri, Aug 24, 2018 at 3:56 PM ramandu...@gmail.com 
wrote:

> Hi All,
> Any pointer on this would be helpful. We have added extra logs and are
> trying few thing to get the root cause. But we are getting logs like "Task
> is not able to run".
> And we are not getting any resource usage related error.
>
> Thanks,
> Raman Gupta
>
> On 2018/08/21 16:46:56, ramandu...@gmail.com 
> wrote:
> > Hi All,
> > As per http://docs.sqlalchemy.org/en/latest/core/connections.html link
> db engine is not portable across process boundaries
> > "For a multiple-process application that uses the os.fork system call,
> or for example the Python multiprocessing module, it’s usually required
> that a separate Engine be used for each child process. This is because the
> Engine maintains a reference to a connection pool that ultimately
> references DBAPI connections - these tend to not be portable across process
> boundaries"
> > Please correct me if I am wrong but It seems that in Airflow 1.9 child
> processes don't create separate DB engine and so there is only one single
> DB Engine which is shared among child processes which might be causing this
> issue.
> >
> > Thanks,
> > Raman Gupta
> >
> >
> > On 2018/08/21 15:41:14, raman gupta  wrote:
> > > One possibility is the unavailability of session while calling
> > > self.task_instance._check_and_change_state_before_execution
> > > function.
> > > (Session is provided via @provide_session decorator)
> > >
> > > On Tue, Aug 21, 2018 at 7:09 PM vardangupta...@gmail.com <
> > > vardangupta...@gmail.com> wrote:
> > >
> > > > Is there any possibility that on call of function
> > > > _check_and_change_state_before_execution at
> > > >
> https://github.com/apache/incubator-airflow/blob/v1-9-stable/airflow/jobs.py#L2500
> ,
> > > > this method is not actually being called
> > > >
> https://github.com/apache/incubator-airflow/blob/v1-9-stable/airflow/models.py#L1299
> ?
> > > > Because even in a happy scenario, no logs is printed from method's
> > > > implementation and directly control is reaching here
> > > >
> https://github.com/apache/incubator-airflow/blob/v1-9-stable/airflow/jobs.py#L2512
> > > > while in stuck phase, we are seeing this log
> > > >
> https://github.com/apache/incubator-airflow/blob/v1-9-stable/airflow/jobs.py#L2508
> > > > i.e. Task is not able to be run, FYI we've not set any sort of
> dependency
> > > > with dag.
> > > >
> > > > Regards,
> > > > Vardan Gupta
> > > >
> > > > On 2018/08/16 08:25:37, ramandu...@gmail.com 
> > > > wrote:
> > > > > Hi All,
> > > > >
> > > > > We are using airflow 1.9 with Local Executor more. Intermittently
> we are
> > > > observing that tasks are getting stuck in "up_for_retry" mode and are
> > > > getting retried again and again exceeding their configured max
> retries
> > > > count. like we have configured max retries as 2 but task is retried
> 15
> > > > times and got stuck in up_for_retry state.
> > > > > Any pointer on this would be helpful.
> > > > >
> > > > > Thanks,
> > > > > Raman Gupta
> > > > >
> > > >
> > >
> >
>

Re: invite me apache-airflow.slack.com

[Bas Harenslak]

You can register with your own email address at 
https://apache-airflow-slack.herokuapp.com


On 3 Oct 2018, at 06:40, 김용휘 mailto:fkl...@naver.com>> wrote:

If you are admin below the slack-channel.
please invite me. apache-airflow.slack.
fkl...@naver.com