Re: How to bes use Google Cloud Storage for logging?

[Feng Lu]

Hi Kevin,

Kindly see my reply inline:

On Mon, Dec 18, 2017 at 3:28 PM, Kevin Lam  wrote:

> Hi,
>
> I'm trying to get airflow to use GCS for logging purposes and had a few
> questions.
>
> We're currently using Airflow 1.9rc2, running in a Kubernetes Airflow
> deployment (similar to https://github.com/mumoshu/kube-airflow)
>
> 1/ Seems like the logging code has been going through some changes in the
> recent versions. What's the correct way to set up GCS for logging? Is it by
> just specifying remote_base_log_folder and remote_log_conn_id in
> airflow.cfg? Or by following this guide:
> http://airflow.readthedocs.io/en/latest/integration.html#gcp, using the
> python based logging config? Is there an Airflow version that we should use
> to be most stable?
>
The python based logging config is the right place to make changes, in our
test setup, we override the airflow_local_settings.py similarly to the link
you pasted.
You may also want to config: [core]task_log_reader = gcs.task


>
> 2/ Is there a way to encode the connection for GCS in a file so that one
> doesn't have to open the webserver and create it from the admin panel? It'd
> be nice if the GCS connection would be automatically created.
>
Unfortunately GCS connection ties to some GCP project and is impossible to
pre-populate.
Airflow1.9 should fix the gcp connection type issue  (
https://github.com/apache/incubator-airflow/commit/2f107d8a30910fd025774004d5c4c95407ed55c5),
so you can use airflow connections CLI directly.


>
> Thanks in advance for your help!
>

How to bes use Google Cloud Storage for logging?

[Kevin Lam]

Hi,

I'm trying to get airflow to use GCS for logging purposes and had a few
questions.

We're currently using Airflow 1.9rc2, running in a Kubernetes Airflow
deployment (similar to https://github.com/mumoshu/kube-airflow)

1/ Seems like the logging code has been going through some changes in the
recent versions. What's the correct way to set up GCS for logging? Is it by
just specifying remote_base_log_folder and remote_log_conn_id in
airflow.cfg? Or by following this guide:
http://airflow.readthedocs.io/en/latest/integration.html#gcp, using the
python based logging config? Is there an Airflow version that we should use
to be most stable?

2/ Is there a way to encode the connection for GCS in a file so that one
doesn't have to open the webserver and create it from the admin panel? It'd
be nice if the GCS connection would be automatically created.

Thanks in advance for your help!

Re: [RESULT][VOTE] Airflow 1.9.0rc8

[Chris Riccomini]

Thanks to all for helping validate it. :)

On Mon, Dec 18, 2017 at 12:48 PM, Ash Berlin-Taylor <
ash_airflowl...@firemirror.com> wrote:

> Woot! Great news, and thanks for all the time you've spent preparing all
> the RCs.
>
> -ash
>
> > On 18 Dec 2017, at 20:13, Chris Riccomini  wrote:
> >
> > Hello,
> >
> > Apache Airflow (incubating) 1.9.0 (based on RC8) has been accepted.
> >
> > 5 “+1” binding votes received:
> >
> > - Maxime Beauchemin (binding)
> > - Chris Riccomini (binding)
> > - Bolke de Bruin (binding)
> > - Joy Gao (binding)
> > - Fokko Driesprong (binding)
> >
> > My next step is to open a thread with the IPMC.
> >
> > Cheers,
> > Chris
> >
> > On Mon, Dec 18, 2017 at 11:26 AM, Chris Riccomini  >
> > wrote:
> >
> >> I believe so, yes.
> >>
> >> I will proceed with release. :)
> >>
> >> On Mon, Dec 18, 2017 at 11:20 AM, Bolke de Bruin 
> >> wrote:
> >>
> >>> But it doesn’t stop us from releasing now right? You can just update
> the
> >>> tag. It is not connected to the artifacts.
> >>>
> >>> Bolke.
> >>>
> >>> Sent from my iPhone
> >>>
>  On 18 Dec 2017, at 18:51, Chris Riccomini 
> >>> wrote:
> 
>  https://github.com/apache/incubator-airflow/pull/2889
> 
>  On Mon, Dec 18, 2017 at 9:44 AM, Chris Riccomini <
> criccom...@apache.org
> 
>  wrote:
> 
> > Sigh. There's a catch 22 here. If we tag an RC as 1.9.0, we'll have
> to
> > keep editing the tag every time there's a new RC. I agree that
> >>> removing the
> > check seems like the best course of action.
> >
> > On Sun, Dec 17, 2017 at 12:01 AM, Bolke de Bruin 
> > wrote:
> >
> >> This is just matter of setting the tag in the repo right?
> >>
> >> We should remove that check or make it not fail at least. It is
> >> ridiculous.
> >>
> >> B.
> >>
> >> Verstuurd vanaf mijn iPad
> >>
> >>> Op 17 dec. 2017 om 07:32 heeft Joy Gao  het
> volgende
> >> geschreven:
> >>>
> >>> Ahh, tested the build on a fresh virtualenv, which succeeded *the
> >>> first
> >>> time* given gitpython was not installed and it skipped the
> assertion
> >> check
> >>>  >> setup.py#L68-L73>.
> >>> Build fails on re-installs :(
> >>>
>  On Fri, Dec 15, 2017 at 6:01 PM, Feng Lu
>  
> >> wrote:
> 
>  +0.5 (non-binding)
> 
>  Looks like the version(1.9.0) and tag(1.9.0rc8) is mismatched,
>   >> setup.py#L87>
>  which will cause the installation (pip install or python setup) to
> >> error
>  out and fail.
>  nit: mind also updating the release log "
>  https://github.com/apache/incubator-airflow/blob/1.9.0rc8/
> >> CHANGELOG.txt"
> 
> 
>  On Fri, Dec 15, 2017 at 3:21 PM, Driesprong, Fokko
> >> 
>  wrote:
> 
> > +1 binding
> >
> > Op vr 15 dec. 2017 om 23:39 schreef Bolke de Bruin <
> >>> bdbr...@gmail.com
> >>>
> >
> >> +1, binding
> >>
> >> Checked sigs, version, source is there (did not check build),
> bin
> >>> is
> > there.
> >>
> >> Bolke
> >>
> >> Verstuurd vanaf mijn iPad
> >>
> >>> Op 15 dec. 2017 om 23:31 heeft Joy Gao  het
> >> volgende
> >> geschreven:
> >>>
> >>> +1, binding
> >>>
> >>> Thank you Chris!
> >>>
> >>> On Fri, Dec 15, 2017 at 2:30 PM, Chris Riccomini <
> > criccom...@apache.org>
> >>> wrote:
> >>>
>  Hey all,
> 
>  (Last time, I hope)^2
> 
>  I have cut Airflow 1.9.0 RC8. This email is calling a vote on
> >>> the
> >> release,
>  which will last for 72 hours. Consider this my (binding) +1.
> 
>  Airflow 1.9.0 RC8 is available at:
> 
>  https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9
> >>> .0rc8/
> 
>  apache-airflow-1.9.0rc8+incubating-source.tar.gz is a source
>  release
> >> that
>  comes with INSTALL instructions.
>  apache-airflow-1.9.0rc8+incubating-bin.tar.gz is the binary
> >>> Python
> >> "sdist"
>  release.
> 
>  Public keys are available at:
> 
>  https://dist.apache.org/repos/dist/release/incubator/airflow/
> 
>  The release contains no new JIRAs. Just a version fix.
> 
>  I also had to change the version number to exclude the `rc6`
> >>> string
>  as
> >> well
>  as the "+incubating" string, so it's now simply 1.9.0. This
> will
>  allow
> >> us
>  to rename the artifact without modify

Re: [RESULT][VOTE] Airflow 1.9.0rc8

[Ash Berlin-Taylor]

Woot! Great news, and thanks for all the time you've spent preparing all the 
RCs.

-ash

> On 18 Dec 2017, at 20:13, Chris Riccomini  wrote:
> 
> Hello,
> 
> Apache Airflow (incubating) 1.9.0 (based on RC8) has been accepted.
> 
> 5 “+1” binding votes received:
> 
> - Maxime Beauchemin (binding)
> - Chris Riccomini (binding)
> - Bolke de Bruin (binding)
> - Joy Gao (binding)
> - Fokko Driesprong (binding)
> 
> My next step is to open a thread with the IPMC.
> 
> Cheers,
> Chris
> 
> On Mon, Dec 18, 2017 at 11:26 AM, Chris Riccomini 
> wrote:
> 
>> I believe so, yes.
>> 
>> I will proceed with release. :)
>> 
>> On Mon, Dec 18, 2017 at 11:20 AM, Bolke de Bruin 
>> wrote:
>> 
>>> But it doesn’t stop us from releasing now right? You can just update the
>>> tag. It is not connected to the artifacts.
>>> 
>>> Bolke.
>>> 
>>> Sent from my iPhone
>>> 
 On 18 Dec 2017, at 18:51, Chris Riccomini 
>>> wrote:
 
 https://github.com/apache/incubator-airflow/pull/2889
 
 On Mon, Dec 18, 2017 at 9:44 AM, Chris Riccomini >>> 
 wrote:
 
> Sigh. There's a catch 22 here. If we tag an RC as 1.9.0, we'll have to
> keep editing the tag every time there's a new RC. I agree that
>>> removing the
> check seems like the best course of action.
> 
> On Sun, Dec 17, 2017 at 12:01 AM, Bolke de Bruin 
> wrote:
> 
>> This is just matter of setting the tag in the repo right?
>> 
>> We should remove that check or make it not fail at least. It is
>> ridiculous.
>> 
>> B.
>> 
>> Verstuurd vanaf mijn iPad
>> 
>>> Op 17 dec. 2017 om 07:32 heeft Joy Gao  het volgende
>> geschreven:
>>> 
>>> Ahh, tested the build on a fresh virtualenv, which succeeded *the
>>> first
>>> time* given gitpython was not installed and it skipped the assertion
>> check
>>> > setup.py#L68-L73>.
>>> Build fails on re-installs :(
>>> 
 On Fri, Dec 15, 2017 at 6:01 PM, Feng Lu >>> 
>> wrote:
 
 +0.5 (non-binding)
 
 Looks like the version(1.9.0) and tag(1.9.0rc8) is mismatched,
 > setup.py#L87>
 which will cause the installation (pip install or python setup) to
>> error
 out and fail.
 nit: mind also updating the release log "
 https://github.com/apache/incubator-airflow/blob/1.9.0rc8/
>> CHANGELOG.txt"
 
 
 On Fri, Dec 15, 2017 at 3:21 PM, Driesprong, Fokko
>> 
 wrote:
 
> +1 binding
> 
> Op vr 15 dec. 2017 om 23:39 schreef Bolke de Bruin <
>>> bdbr...@gmail.com
>>> 
> 
>> +1, binding
>> 
>> Checked sigs, version, source is there (did not check build), bin
>>> is
> there.
>> 
>> Bolke
>> 
>> Verstuurd vanaf mijn iPad
>> 
>>> Op 15 dec. 2017 om 23:31 heeft Joy Gao  het
>> volgende
>> geschreven:
>>> 
>>> +1, binding
>>> 
>>> Thank you Chris!
>>> 
>>> On Fri, Dec 15, 2017 at 2:30 PM, Chris Riccomini <
> criccom...@apache.org>
>>> wrote:
>>> 
 Hey all,
 
 (Last time, I hope)^2
 
 I have cut Airflow 1.9.0 RC8. This email is calling a vote on
>>> the
>> release,
 which will last for 72 hours. Consider this my (binding) +1.
 
 Airflow 1.9.0 RC8 is available at:
 
 https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9
>>> .0rc8/
 
 apache-airflow-1.9.0rc8+incubating-source.tar.gz is a source
 release
>> that
 comes with INSTALL instructions.
 apache-airflow-1.9.0rc8+incubating-bin.tar.gz is the binary
>>> Python
>> "sdist"
 release.
 
 Public keys are available at:
 
 https://dist.apache.org/repos/dist/release/incubator/airflow/
 
 The release contains no new JIRAs. Just a version fix.
 
 I also had to change the version number to exclude the `rc6`
>>> string
 as
>> well
 as the "+incubating" string, so it's now simply 1.9.0. This will
 allow
>> us
 to rename the artifact without modifying the artifact checksums
>> when
> we
 actually release.
 
 See JIRAs that were in 1.9.0RC7 and before (see previous VOTE
>>> email
> for
 full list).
 
 Cheers,
 Chris
 
>> 
> 
 
>> 
> 
> 
>>> 
>> 
>> 

[RESULT][VOTE] Airflow 1.9.0rc8

[Chris Riccomini]

Hello,

Apache Airflow (incubating) 1.9.0 (based on RC8) has been accepted.

5 “+1” binding votes received:

- Maxime Beauchemin (binding)
- Chris Riccomini (binding)
- Bolke de Bruin (binding)
- Joy Gao (binding)
- Fokko Driesprong (binding)

My next step is to open a thread with the IPMC.

Cheers,
Chris

On Mon, Dec 18, 2017 at 11:26 AM, Chris Riccomini 
wrote:

> I believe so, yes.
>
> I will proceed with release. :)
>
> On Mon, Dec 18, 2017 at 11:20 AM, Bolke de Bruin 
> wrote:
>
>> But it doesn’t stop us from releasing now right? You can just update the
>> tag. It is not connected to the artifacts.
>>
>> Bolke.
>>
>> Sent from my iPhone
>>
>> > On 18 Dec 2017, at 18:51, Chris Riccomini 
>> wrote:
>> >
>> > https://github.com/apache/incubator-airflow/pull/2889
>> >
>> > On Mon, Dec 18, 2017 at 9:44 AM, Chris Riccomini > >
>> > wrote:
>> >
>> >> Sigh. There's a catch 22 here. If we tag an RC as 1.9.0, we'll have to
>> >> keep editing the tag every time there's a new RC. I agree that
>> removing the
>> >> check seems like the best course of action.
>> >>
>> >> On Sun, Dec 17, 2017 at 12:01 AM, Bolke de Bruin 
>> >> wrote:
>> >>
>> >>> This is just matter of setting the tag in the repo right?
>> >>>
>> >>> We should remove that check or make it not fail at least. It is
>> >>> ridiculous.
>> >>>
>> >>> B.
>> >>>
>> >>> Verstuurd vanaf mijn iPad
>> >>>
>>  Op 17 dec. 2017 om 07:32 heeft Joy Gao  het volgende
>> >>> geschreven:
>> 
>>  Ahh, tested the build on a fresh virtualenv, which succeeded *the
>> first
>>  time* given gitpython was not installed and it skipped the assertion
>> >>> check
>>  > >>> setup.py#L68-L73>.
>>  Build fails on re-installs :(
>> 
>> > On Fri, Dec 15, 2017 at 6:01 PM, Feng Lu > >
>> >>> wrote:
>> >
>> > +0.5 (non-binding)
>> >
>> > Looks like the version(1.9.0) and tag(1.9.0rc8) is mismatched,
>> > > >>> setup.py#L87>
>> > which will cause the installation (pip install or python setup) to
>> >>> error
>> > out and fail.
>> > nit: mind also updating the release log "
>> > https://github.com/apache/incubator-airflow/blob/1.9.0rc8/
>> >>> CHANGELOG.txt"
>> >
>> >
>> > On Fri, Dec 15, 2017 at 3:21 PM, Driesprong, Fokko
>> >>> 
>> > wrote:
>> >
>> >> +1 binding
>> >>
>> >> Op vr 15 dec. 2017 om 23:39 schreef Bolke de Bruin <
>> bdbr...@gmail.com
>> 
>> >>
>> >>> +1, binding
>> >>>
>> >>> Checked sigs, version, source is there (did not check build), bin
>> is
>> >> there.
>> >>>
>> >>> Bolke
>> >>>
>> >>> Verstuurd vanaf mijn iPad
>> >>>
>>  Op 15 dec. 2017 om 23:31 heeft Joy Gao  het
>> >>> volgende
>> >>> geschreven:
>> 
>>  +1, binding
>> 
>>  Thank you Chris!
>> 
>>  On Fri, Dec 15, 2017 at 2:30 PM, Chris Riccomini <
>> >> criccom...@apache.org>
>>  wrote:
>> 
>> > Hey all,
>> >
>> > (Last time, I hope)^2
>> >
>> > I have cut Airflow 1.9.0 RC8. This email is calling a vote on
>> the
>> >>> release,
>> > which will last for 72 hours. Consider this my (binding) +1.
>> >
>> > Airflow 1.9.0 RC8 is available at:
>> >
>> > https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9
>> .0rc8/
>> >
>> > apache-airflow-1.9.0rc8+incubating-source.tar.gz is a source
>> > release
>> >>> that
>> > comes with INSTALL instructions.
>> > apache-airflow-1.9.0rc8+incubating-bin.tar.gz is the binary
>> Python
>> >>> "sdist"
>> > release.
>> >
>> > Public keys are available at:
>> >
>> > https://dist.apache.org/repos/dist/release/incubator/airflow/
>> >
>> > The release contains no new JIRAs. Just a version fix.
>> >
>> > I also had to change the version number to exclude the `rc6`
>> string
>> > as
>> >>> well
>> > as the "+incubating" string, so it's now simply 1.9.0. This will
>> > allow
>> >>> us
>> > to rename the artifact without modifying the artifact checksums
>> >>> when
>> >> we
>> > actually release.
>> >
>> > See JIRAs that were in 1.9.0RC7 and before (see previous VOTE
>> email
>> >> for
>> > full list).
>> >
>> > Cheers,
>> > Chris
>> >
>> >>>
>> >>
>> >
>> >>>
>> >>
>> >>
>>
>
>

Re: Airflow 2.0

[Bolke de Bruin]

Hi George,

It does make sense to keep SQLite on the other hand a Docker image with all the 
components might just be as convenient?

Dropping stuff would lessen the burden of maintenance. There is a lot of cruft 
that is not used. This reduces the surface for bugs and makes it easier to do 
the optimizations you refer too. However, I’m not married to a 2.0 release now, 
but it does look fresh ;-).

Bolke

Verstuurd vanaf mijn iPad

> Op 18 dec. 2017 om 19:48 heeft George Leslie-Waksman 
>  het volgende geschreven:
> 
> I really do not think we should drop sqlite support. We use sqlite for
> local testing/development; I used it when doing my initial evaluation of
> Airflow; and there are regular comments in this group about people using
> sqlite locally for their workflows. It feels like a critical feature for a
> lot of use cases.
> 
> I also feel like we're kind of jumping the gun on pushing for 2.0 (is it
> just because 9 is the last single digit?). There is a lot of the code base
> that's pretty hairy and a number of things that are fairly non-performant /
> buggy. I don't think that we NEED to break backward compatibility to clean
> things up and fix things. I don't particularly like pickle or the old
> import style but I don't feel like supporting them is terribly onerous (I
> could be wrong).
> 
> New webserver, releasing the API, and even timezones are all things that
> can be released in a non-breaking fashion.
> 
> --George
> 
>> On Thu, Dec 14, 2017 at 2:20 PM Alek Storm  wrote:
>> 
>> We use sqlite for developing Airflow DAGs locally, and doing basic checks
>> for syntax/import errors in our CI/CD pipeline.
>> 
>> Alek
>> 
>> On Thu, Dec 14, 2017 at 4:19 PM, Chris Riccomini 
>> wrote:
>> 
>>> Agree with Bolke that it's a good idea, but major work. I will push back
>> on
>>> this for 2.0 mostly due to time concerns. I don't want 2.0 to take months
>>> and months to get out.
>>> 
 Btw: what about dropping sqlite support?
>>> 
>>> I'm fine with this, but isn't this really useful for demos? Seems to me
>>> that the out of the box experience from airflow is one of the things that
>>> make sit so compelling.
>>> 
>>> On Thu, Dec 14, 2017 at 2:12 PM, Bolke de Bruin 
>> wrote:
>>> 
 That will take quite some work. It is a a good idea but also a major
 change. Not sure if we should target that.
 
 Btw: what about dropping sqlite support?
 
 Verstuurd vanaf mijn iPad
 
> Op 14 dec. 2017 om 21:19 heeft Gael Magnan 
>> het
 volgende geschreven:
> 
> Hi,
> 
> haven't been following much lately but on the import side of things,
 isn't
> Airflow 2 the best moment to change to a pip plugin system for
>> imports
>>> of
> third party stuff?
> I.E being able to add a new type of credentials, operator etc..
>> without
> touching to the airflow code itself or having them in a special
>> folder.
> 
> Regards
> Gael
> 
> 
> 
> Le jeu. 14 déc. 2017 à 14:17, Chris Riccomini >> 
>>> a
> écrit :
> 
>> I'm fine with sensor refactor. Added to Wiki.
>> 
>> On Thu, Dec 14, 2017 at 11:16 AM, Chris Riccomini <
 criccom...@apache.org>
>> wrote:
>> 
>>> @Bolke,
>>> 
 Should we, before 2.0, start the graduation from the incubator?
>>> 
>>> No, I'd rather keep them separate. We can certainly start
>> graduation,
 but
>>> I don't want to block 2.0. Can pursue them in parallel.
>>> 
>>> On Thu, Dec 14, 2017 at 11:15 AM, Andy Hadjigeorgiou <
>> andyxha...@gmail.com
 wrote:
>>> 
 Does it make sense to include sensors.py refactor in 2.0, so we
>> can
 retire the old import structure easily and support the new sensors
>> package
 import structure?
 
 - Andy
 
 On Thu, Dec 14, 2017 at 2:12 PM, Driesprong, Fokko
 >> 
 wrote:
 
> Hi all,
> 
> Good initiative. I would be happy to refactor the sensors
>> package.
>>> I
> started on it but it changes a lot, all the imports will break.
> 
> https://github.com/apache/incubator-airflow/pull/2875
> 
> What do you guys think?
> 
> Cheers, Fokko
> 
> 2017-12-14 20:09 GMT+01:00 Chris Riccomini <
>> criccom...@apache.org
 :
> 
>> I have created a wiki here:
>> 
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0
>> 
>> To track features and progress.
>> 
>> On Thu, Dec 14, 2017 at 11:08 AM, Chris Riccomini <
> criccom...@apache.org>
>> wrote:
>> 
 Re: #2: Is there a current ticket out for removing the legacy
> import
>>> style?
>>> 
>>> No, I don't think so, but you can create one! :)
>>> 
>>> On Thu, Dec 14, 2017 at 11:06 AM, Andy Hadjigeorgiou <
>> andyxha...@gm

Re: [VOTE] Airflow 1.9.0rc8

[Chris Riccomini]

I believe so, yes.

I will proceed with release. :)

On Mon, Dec 18, 2017 at 11:20 AM, Bolke de Bruin  wrote:

> But it doesn’t stop us from releasing now right? You can just update the
> tag. It is not connected to the artifacts.
>
> Bolke.
>
> Sent from my iPhone
>
> > On 18 Dec 2017, at 18:51, Chris Riccomini  wrote:
> >
> > https://github.com/apache/incubator-airflow/pull/2889
> >
> > On Mon, Dec 18, 2017 at 9:44 AM, Chris Riccomini 
> > wrote:
> >
> >> Sigh. There's a catch 22 here. If we tag an RC as 1.9.0, we'll have to
> >> keep editing the tag every time there's a new RC. I agree that removing
> the
> >> check seems like the best course of action.
> >>
> >> On Sun, Dec 17, 2017 at 12:01 AM, Bolke de Bruin 
> >> wrote:
> >>
> >>> This is just matter of setting the tag in the repo right?
> >>>
> >>> We should remove that check or make it not fail at least. It is
> >>> ridiculous.
> >>>
> >>> B.
> >>>
> >>> Verstuurd vanaf mijn iPad
> >>>
>  Op 17 dec. 2017 om 07:32 heeft Joy Gao  het volgende
> >>> geschreven:
> 
>  Ahh, tested the build on a fresh virtualenv, which succeeded *the
> first
>  time* given gitpython was not installed and it skipped the assertion
> >>> check
>   >>> setup.py#L68-L73>.
>  Build fails on re-installs :(
> 
> > On Fri, Dec 15, 2017 at 6:01 PM, Feng Lu 
> >>> wrote:
> >
> > +0.5 (non-binding)
> >
> > Looks like the version(1.9.0) and tag(1.9.0rc8) is mismatched,
> >  >>> setup.py#L87>
> > which will cause the installation (pip install or python setup) to
> >>> error
> > out and fail.
> > nit: mind also updating the release log "
> > https://github.com/apache/incubator-airflow/blob/1.9.0rc8/
> >>> CHANGELOG.txt"
> >
> >
> > On Fri, Dec 15, 2017 at 3:21 PM, Driesprong, Fokko
> >>> 
> > wrote:
> >
> >> +1 binding
> >>
> >> Op vr 15 dec. 2017 om 23:39 schreef Bolke de Bruin <
> bdbr...@gmail.com
> 
> >>
> >>> +1, binding
> >>>
> >>> Checked sigs, version, source is there (did not check build), bin
> is
> >> there.
> >>>
> >>> Bolke
> >>>
> >>> Verstuurd vanaf mijn iPad
> >>>
>  Op 15 dec. 2017 om 23:31 heeft Joy Gao  het
> >>> volgende
> >>> geschreven:
> 
>  +1, binding
> 
>  Thank you Chris!
> 
>  On Fri, Dec 15, 2017 at 2:30 PM, Chris Riccomini <
> >> criccom...@apache.org>
>  wrote:
> 
> > Hey all,
> >
> > (Last time, I hope)^2
> >
> > I have cut Airflow 1.9.0 RC8. This email is calling a vote on the
> >>> release,
> > which will last for 72 hours. Consider this my (binding) +1.
> >
> > Airflow 1.9.0 RC8 is available at:
> >
> > https://dist.apache.org/repos/dist/dev/incubator/airflow/1.
> 9.0rc8/
> >
> > apache-airflow-1.9.0rc8+incubating-source.tar.gz is a source
> > release
> >>> that
> > comes with INSTALL instructions.
> > apache-airflow-1.9.0rc8+incubating-bin.tar.gz is the binary
> Python
> >>> "sdist"
> > release.
> >
> > Public keys are available at:
> >
> > https://dist.apache.org/repos/dist/release/incubator/airflow/
> >
> > The release contains no new JIRAs. Just a version fix.
> >
> > I also had to change the version number to exclude the `rc6`
> string
> > as
> >>> well
> > as the "+incubating" string, so it's now simply 1.9.0. This will
> > allow
> >>> us
> > to rename the artifact without modifying the artifact checksums
> >>> when
> >> we
> > actually release.
> >
> > See JIRAs that were in 1.9.0RC7 and before (see previous VOTE
> email
> >> for
> > full list).
> >
> > Cheers,
> > Chris
> >
> >>>
> >>
> >
> >>>
> >>
> >>
>

Re: [VOTE] Airflow 1.9.0rc8

[Bolke de Bruin]

But it doesn’t stop us from releasing now right? You can just update the tag. 
It is not connected to the artifacts. 

Bolke. 

Sent from my iPhone

> On 18 Dec 2017, at 18:51, Chris Riccomini  wrote:
> 
> https://github.com/apache/incubator-airflow/pull/2889
> 
> On Mon, Dec 18, 2017 at 9:44 AM, Chris Riccomini 
> wrote:
> 
>> Sigh. There's a catch 22 here. If we tag an RC as 1.9.0, we'll have to
>> keep editing the tag every time there's a new RC. I agree that removing the
>> check seems like the best course of action.
>> 
>> On Sun, Dec 17, 2017 at 12:01 AM, Bolke de Bruin 
>> wrote:
>> 
>>> This is just matter of setting the tag in the repo right?
>>> 
>>> We should remove that check or make it not fail at least. It is
>>> ridiculous.
>>> 
>>> B.
>>> 
>>> Verstuurd vanaf mijn iPad
>>> 
 Op 17 dec. 2017 om 07:32 heeft Joy Gao  het volgende
>>> geschreven:
 
 Ahh, tested the build on a fresh virtualenv, which succeeded *the first
 time* given gitpython was not installed and it skipped the assertion
>>> check
 >> setup.py#L68-L73>.
 Build fails on re-installs :(
 
> On Fri, Dec 15, 2017 at 6:01 PM, Feng Lu 
>>> wrote:
> 
> +0.5 (non-binding)
> 
> Looks like the version(1.9.0) and tag(1.9.0rc8) is mismatched,
> >> setup.py#L87>
> which will cause the installation (pip install or python setup) to
>>> error
> out and fail.
> nit: mind also updating the release log "
> https://github.com/apache/incubator-airflow/blob/1.9.0rc8/
>>> CHANGELOG.txt"
> 
> 
> On Fri, Dec 15, 2017 at 3:21 PM, Driesprong, Fokko
>>> 
> wrote:
> 
>> +1 binding
>> 
>> Op vr 15 dec. 2017 om 23:39 schreef Bolke de Bruin >>> 
>> 
>>> +1, binding
>>> 
>>> Checked sigs, version, source is there (did not check build), bin is
>> there.
>>> 
>>> Bolke
>>> 
>>> Verstuurd vanaf mijn iPad
>>> 
 Op 15 dec. 2017 om 23:31 heeft Joy Gao  het
>>> volgende
>>> geschreven:
 
 +1, binding
 
 Thank you Chris!
 
 On Fri, Dec 15, 2017 at 2:30 PM, Chris Riccomini <
>> criccom...@apache.org>
 wrote:
 
> Hey all,
> 
> (Last time, I hope)^2
> 
> I have cut Airflow 1.9.0 RC8. This email is calling a vote on the
>>> release,
> which will last for 72 hours. Consider this my (binding) +1.
> 
> Airflow 1.9.0 RC8 is available at:
> 
> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9.0rc8/
> 
> apache-airflow-1.9.0rc8+incubating-source.tar.gz is a source
> release
>>> that
> comes with INSTALL instructions.
> apache-airflow-1.9.0rc8+incubating-bin.tar.gz is the binary Python
>>> "sdist"
> release.
> 
> Public keys are available at:
> 
> https://dist.apache.org/repos/dist/release/incubator/airflow/
> 
> The release contains no new JIRAs. Just a version fix.
> 
> I also had to change the version number to exclude the `rc6` string
> as
>>> well
> as the "+incubating" string, so it's now simply 1.9.0. This will
> allow
>>> us
> to rename the artifact without modifying the artifact checksums
>>> when
>> we
> actually release.
> 
> See JIRAs that were in 1.9.0RC7 and before (see previous VOTE email
>> for
> full list).
> 
> Cheers,
> Chris
> 
>>> 
>> 
> 
>>> 
>> 
>> 

Re: Airflow 2.0

[George Leslie-Waksman]

I really do not think we should drop sqlite support. We use sqlite for
local testing/development; I used it when doing my initial evaluation of
Airflow; and there are regular comments in this group about people using
sqlite locally for their workflows. It feels like a critical feature for a
lot of use cases.

I also feel like we're kind of jumping the gun on pushing for 2.0 (is it
just because 9 is the last single digit?). There is a lot of the code base
that's pretty hairy and a number of things that are fairly non-performant /
buggy. I don't think that we NEED to break backward compatibility to clean
things up and fix things. I don't particularly like pickle or the old
import style but I don't feel like supporting them is terribly onerous (I
could be wrong).

New webserver, releasing the API, and even timezones are all things that
can be released in a non-breaking fashion.

--George

On Thu, Dec 14, 2017 at 2:20 PM Alek Storm  wrote:

> We use sqlite for developing Airflow DAGs locally, and doing basic checks
> for syntax/import errors in our CI/CD pipeline.
>
> Alek
>
> On Thu, Dec 14, 2017 at 4:19 PM, Chris Riccomini 
> wrote:
>
> > Agree with Bolke that it's a good idea, but major work. I will push back
> on
> > this for 2.0 mostly due to time concerns. I don't want 2.0 to take months
> > and months to get out.
> >
> > > Btw: what about dropping sqlite support?
> >
> > I'm fine with this, but isn't this really useful for demos? Seems to me
> > that the out of the box experience from airflow is one of the things that
> > make sit so compelling.
> >
> > On Thu, Dec 14, 2017 at 2:12 PM, Bolke de Bruin 
> wrote:
> >
> > > That will take quite some work. It is a a good idea but also a major
> > > change. Not sure if we should target that.
> > >
> > > Btw: what about dropping sqlite support?
> > >
> > > Verstuurd vanaf mijn iPad
> > >
> > > > Op 14 dec. 2017 om 21:19 heeft Gael Magnan 
> het
> > > volgende geschreven:
> > > >
> > > > Hi,
> > > >
> > > > haven't been following much lately but on the import side of things,
> > > isn't
> > > > Airflow 2 the best moment to change to a pip plugin system for
> imports
> > of
> > > > third party stuff?
> > > > I.E being able to add a new type of credentials, operator etc..
> without
> > > > touching to the airflow code itself or having them in a special
> folder.
> > > >
> > > > Regards
> > > > Gael
> > > >
> > > >
> > > >
> > > > Le jeu. 14 déc. 2017 à 14:17, Chris Riccomini  >
> > a
> > > > écrit :
> > > >
> > > >> I'm fine with sensor refactor. Added to Wiki.
> > > >>
> > > >> On Thu, Dec 14, 2017 at 11:16 AM, Chris Riccomini <
> > > criccom...@apache.org>
> > > >> wrote:
> > > >>
> > > >>> @Bolke,
> > > >>>
> > >  Should we, before 2.0, start the graduation from the incubator?
> > > >>>
> > > >>> No, I'd rather keep them separate. We can certainly start
> graduation,
> > > but
> > > >>> I don't want to block 2.0. Can pursue them in parallel.
> > > >>>
> > > >>> On Thu, Dec 14, 2017 at 11:15 AM, Andy Hadjigeorgiou <
> > > >> andyxha...@gmail.com
> > >  wrote:
> > > >>>
> > >  Does it make sense to include sensors.py refactor in 2.0, so we
> can
> > >  retire the old import structure easily and support the new sensors
> > > >> package
> > >  import structure?
> > > 
> > >  - Andy
> > > 
> > >  On Thu, Dec 14, 2017 at 2:12 PM, Driesprong, Fokko
> > >  > > >>>
> > >  wrote:
> > > 
> > > > Hi all,
> > > >
> > > > Good initiative. I would be happy to refactor the sensors
> package.
> > I
> > > > started on it but it changes a lot, all the imports will break.
> > > >
> > > > https://github.com/apache/incubator-airflow/pull/2875
> > > >
> > > > What do you guys think?
> > > >
> > > > Cheers, Fokko
> > > >
> > > > 2017-12-14 20:09 GMT+01:00 Chris Riccomini <
> criccom...@apache.org
> > >:
> > > >
> > > >> I have created a wiki here:
> > > >>
> > > >> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0
> > > >>
> > > >> To track features and progress.
> > > >>
> > > >> On Thu, Dec 14, 2017 at 11:08 AM, Chris Riccomini <
> > > > criccom...@apache.org>
> > > >> wrote:
> > > >>
> > >  Re: #2: Is there a current ticket out for removing the legacy
> > > > import
> > > >>> style?
> > > >>>
> > > >>> No, I don't think so, but you can create one! :)
> > > >>>
> > > >>> On Thu, Dec 14, 2017 at 11:06 AM, Andy Hadjigeorgiou <
> > > >> andyxha...@gmail.com
> > >  wrote:
> > > >>>
> > >  This sounds great, something I'd like to see updated for 2.0
> > > > release (or
> > >  before) is the Airflow documentation
> > >   (
> > >  http://airflow.readthedocs.io/en/latest/installation.html).
> It
> > > > seems
> > > >> that
> > >  updating the repo does not update this site - and given t

Re: [VOTE] Airflow 1.9.0rc8

[Chris Riccomini]

https://github.com/apache/incubator-airflow/pull/2889

On Mon, Dec 18, 2017 at 9:44 AM, Chris Riccomini 
wrote:

> Sigh. There's a catch 22 here. If we tag an RC as 1.9.0, we'll have to
> keep editing the tag every time there's a new RC. I agree that removing the
> check seems like the best course of action.
>
> On Sun, Dec 17, 2017 at 12:01 AM, Bolke de Bruin 
> wrote:
>
>> This is just matter of setting the tag in the repo right?
>>
>> We should remove that check or make it not fail at least. It is
>> ridiculous.
>>
>> B.
>>
>> Verstuurd vanaf mijn iPad
>>
>> > Op 17 dec. 2017 om 07:32 heeft Joy Gao  het volgende
>> geschreven:
>> >
>> > Ahh, tested the build on a fresh virtualenv, which succeeded *the first
>> > time* given gitpython was not installed and it skipped the assertion
>> check
>> > > setup.py#L68-L73>.
>> > Build fails on re-installs :(
>> >
>> >> On Fri, Dec 15, 2017 at 6:01 PM, Feng Lu 
>> wrote:
>> >>
>> >> +0.5 (non-binding)
>> >>
>> >> Looks like the version(1.9.0) and tag(1.9.0rc8) is mismatched,
>> >> > setup.py#L87>
>> >> which will cause the installation (pip install or python setup) to
>> error
>> >> out and fail.
>> >> nit: mind also updating the release log "
>> >> https://github.com/apache/incubator-airflow/blob/1.9.0rc8/
>> CHANGELOG.txt"
>> >>
>> >>
>> >> On Fri, Dec 15, 2017 at 3:21 PM, Driesprong, Fokko
>> 
>> >> wrote:
>> >>
>> >>> +1 binding
>> >>>
>> >>> Op vr 15 dec. 2017 om 23:39 schreef Bolke de Bruin > >
>> >>>
>>  +1, binding
>> 
>>  Checked sigs, version, source is there (did not check build), bin is
>> >>> there.
>> 
>>  Bolke
>> 
>>  Verstuurd vanaf mijn iPad
>> 
>> > Op 15 dec. 2017 om 23:31 heeft Joy Gao  het
>> volgende
>>  geschreven:
>> >
>> > +1, binding
>> >
>> > Thank you Chris!
>> >
>> > On Fri, Dec 15, 2017 at 2:30 PM, Chris Riccomini <
>> >>> criccom...@apache.org>
>> > wrote:
>> >
>> >> Hey all,
>> >>
>> >> (Last time, I hope)^2
>> >>
>> >> I have cut Airflow 1.9.0 RC8. This email is calling a vote on the
>>  release,
>> >> which will last for 72 hours. Consider this my (binding) +1.
>> >>
>> >> Airflow 1.9.0 RC8 is available at:
>> >>
>> >> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9.0rc8/
>> >>
>> >> apache-airflow-1.9.0rc8+incubating-source.tar.gz is a source
>> >> release
>>  that
>> >> comes with INSTALL instructions.
>> >> apache-airflow-1.9.0rc8+incubating-bin.tar.gz is the binary Python
>>  "sdist"
>> >> release.
>> >>
>> >> Public keys are available at:
>> >>
>> >> https://dist.apache.org/repos/dist/release/incubator/airflow/
>> >>
>> >> The release contains no new JIRAs. Just a version fix.
>> >>
>> >> I also had to change the version number to exclude the `rc6` string
>> >> as
>>  well
>> >> as the "+incubating" string, so it's now simply 1.9.0. This will
>> >> allow
>>  us
>> >> to rename the artifact without modifying the artifact checksums
>> when
>> >>> we
>> >> actually release.
>> >>
>> >> See JIRAs that were in 1.9.0RC7 and before (see previous VOTE email
>> >>> for
>> >> full list).
>> >>
>> >> Cheers,
>> >> Chris
>> >>
>> 
>> >>>
>> >>
>>
>
>

Re: [VOTE] Airflow 1.9.0rc8

[Chris Riccomini]

Sigh. There's a catch 22 here. If we tag an RC as 1.9.0, we'll have to keep
editing the tag every time there's a new RC. I agree that removing the
check seems like the best course of action.

On Sun, Dec 17, 2017 at 12:01 AM, Bolke de Bruin  wrote:

> This is just matter of setting the tag in the repo right?
>
> We should remove that check or make it not fail at least. It is ridiculous.
>
> B.
>
> Verstuurd vanaf mijn iPad
>
> > Op 17 dec. 2017 om 07:32 heeft Joy Gao  het volgende
> geschreven:
> >
> > Ahh, tested the build on a fresh virtualenv, which succeeded *the first
> > time* given gitpython was not installed and it skipped the assertion
> check
> >  0rc8/setup.py#L68-L73>.
> > Build fails on re-installs :(
> >
> >> On Fri, Dec 15, 2017 at 6:01 PM, Feng Lu 
> wrote:
> >>
> >> +0.5 (non-binding)
> >>
> >> Looks like the version(1.9.0) and tag(1.9.0rc8) is mismatched,
> >>  >
> >> which will cause the installation (pip install or python setup) to error
> >> out and fail.
> >> nit: mind also updating the release log "
> >> https://github.com/apache/incubator-airflow/blob/1.9.0rc8/CHANGELOG.txt
> "
> >>
> >>
> >> On Fri, Dec 15, 2017 at 3:21 PM, Driesprong, Fokko  >
> >> wrote:
> >>
> >>> +1 binding
> >>>
> >>> Op vr 15 dec. 2017 om 23:39 schreef Bolke de Bruin 
> >>>
>  +1, binding
> 
>  Checked sigs, version, source is there (did not check build), bin is
> >>> there.
> 
>  Bolke
> 
>  Verstuurd vanaf mijn iPad
> 
> > Op 15 dec. 2017 om 23:31 heeft Joy Gao  het volgende
>  geschreven:
> >
> > +1, binding
> >
> > Thank you Chris!
> >
> > On Fri, Dec 15, 2017 at 2:30 PM, Chris Riccomini <
> >>> criccom...@apache.org>
> > wrote:
> >
> >> Hey all,
> >>
> >> (Last time, I hope)^2
> >>
> >> I have cut Airflow 1.9.0 RC8. This email is calling a vote on the
>  release,
> >> which will last for 72 hours. Consider this my (binding) +1.
> >>
> >> Airflow 1.9.0 RC8 is available at:
> >>
> >> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9.0rc8/
> >>
> >> apache-airflow-1.9.0rc8+incubating-source.tar.gz is a source
> >> release
>  that
> >> comes with INSTALL instructions.
> >> apache-airflow-1.9.0rc8+incubating-bin.tar.gz is the binary Python
>  "sdist"
> >> release.
> >>
> >> Public keys are available at:
> >>
> >> https://dist.apache.org/repos/dist/release/incubator/airflow/
> >>
> >> The release contains no new JIRAs. Just a version fix.
> >>
> >> I also had to change the version number to exclude the `rc6` string
> >> as
>  well
> >> as the "+incubating" string, so it's now simply 1.9.0. This will
> >> allow
>  us
> >> to rename the artifact without modifying the artifact checksums when
> >>> we
> >> actually release.
> >>
> >> See JIRAs that were in 1.9.0RC7 and before (see previous VOTE email
> >>> for
> >> full list).
> >>
> >> Cheers,
> >> Chris
> >>
> 
> >>>
> >>
>