Re: [VOTE] Graduate the Apache Airflow as a TLP

[Hitesh Shah]

+1 (binding)

thanks
Hitesh

On Fri, Nov 30, 2018 at 4:05 PM Arthur Wiedmer 
wrote:

> +1 (binding)
>
> On Fri, Nov 30, 2018, 13:33 Jakob Homan 
> > Hey all!
> >
> > Following a very successful DISCUSS[1] regarding graduating Airflow to
> > Top Level Project (TLP) status, I'm starting the official VOTE.
> >
> > Since entering the Incubator in 2016, the community has:
> >* successfully produced 7 releases
> >* added 9 new committers/PPMC members
> >* built a diverse group of committers from multiple different
> employers
> >* had more than 3,300 JIRA tickets opened
> >* completed the project maturity model with positive responses[2]
> >
> > Accordingly, I believe we're ready to graduate and am calling a VOTE
> > on the following graduation resolution.  This VOTE will remain open
> > for at least 72 hours.  If successful, the resolution will be
> > forwarded to the IPMC for its consideration.  If that VOTE is
> > successful, the resolution will be voted upon by the Board at its next
> > monthly meeting.
> >
> > Everyone is encouraged to vote, even if their vote is not binding.
> > We've built a nice community here, let's make sure everyone has their
> > voice heard.
> >
> > Thanks,
> > Jakob
> >
> > [1]
> >
> https://lists.apache.org/thread.html/%3c0a763b0b-7d0d-4353-979a-ac6769eb0...@gmail.com%3E
> > [2]
> > https://cwiki.apache.org/confluence/display/AIRFLOW/Maturity+Evaluation
> >
> > 
> >
> > Establish the Apache Airflow Project
> >
> > WHEREAS, the Board of Directors deems it to be in the best
> > interests of the Foundation and consistent with the
> > Foundation's purpose to establish a Project Management
> > Committee charged with the creation and maintenance of
> > open-source software, for distribution at no charge to
> > the public, related to workflow automation and scheduling
> > that can be used to author and manage data pipelines.
> >
> > NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> > Committee (PMC), to be known as the "Apache Airflow Project",
> > be and hereby is established pursuant to Bylaws of the
> > Foundation; and be it further
> >
> > RESOLVED, that the Apache Airflow Project be and hereby is
> > responsible for the creation and maintenance of software
> > related to workflow automation and scheduling that can be
> > used to author and manage data pipelines; and be it further
> >
> > RESOLVED, that the office of "Vice President, Apache Airflow" be
> > and hereby is created, the person holding such office to
> > serve at the direction of the Board of Directors as the chair
> > of the Apache Airflow Project, and to have primary responsibility
> > for management of the projects within the scope of
> > responsibility of the Apache Airflow Project; and be it further
> >
> > RESOLVED, that the persons listed immediately below be and
> > hereby are appointed to serve as the initial members of the
> > Apache Airflow Project:
> >
> > * Alex Guziel 
> > * Alex Van Boxel 
> > * Arthur Wiedmer 
> > * Ash Berlin-Taylor 
> > * Bolke de Bruin 
> > * Chris Riccomini 
> > * Dan Davydov 
> > * Fokko Driesprong 
> > * Hitesh Shah 
> > * Jakob Homan 
> > * Jeremiah Lowin 
> > * Joy Gao 
> > * Kaxil Naik 
> > * Maxime Beauchemin 
> > * Siddharth Anand 
> > * Sumit Maheshwari 
> > * Tao Feng 
> >
> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Bolke de Bruin
> > be appointed to the office of Vice President, Apache Airflow, to
> > serve in accordance with and subject to the direction of the
> > Board of Directors and the Bylaws of the Foundation until
> > death, resignation, retirement, removal or disqualification,
> > or until a successor is appointed; and be it further
> >
> > RESOLVED, that the initial Apache Airflow PMC be and hereby is
> > tasked with the creation of a set of bylaws intended to
> > encourage open development and increased participation in the
> > Apache Airflow Project; and be it further
> >
> > RESOLVED, that the Apache Airflow Project be and hereby
> > is tasked with the migration and rationalization of the Apache
> > Incubator Airflow podling; and be it further
> >
> > RESOLVED, that all responsibilities pertaining to the Apache
> > Incubator Airflow podling encumbered upon the Apache Incubator
> > Project are hereafter discharged.
> >
>

Re: [DISCUSS] Apache Airflow graduation from the incubator

[Hitesh Shah]

+1. The Airflow community has come a long way since its addition to the
Incubator and I believe it is more than ready to graduate. Based on past
experience around all the "paperwork", I would suggest that the earliest
graduation date would be the January board meeting and working backwards to
plan out filling in the various wikis and triggering the relevant votes.

thanks
Hitesh



On Mon, Nov 26, 2018 at 2:09 PM Bolke de Bruin  wrote:

> Hi Jakob
>
> Thanks for the vote of confidence! That's appreciated.
>
> I linked the maturity model and already did a grading (I think :-) ) in my
> original mail, so one thing less to worry about. It's probably a good idea
> if some of the other committers and contributers also take a look. 2 items
> I'm unsure about.
>
> Cheers
> Bolke
>
>
>
> Sent from my iPhone
>
> > On 26 Nov 2018, at 22:30, Jakob Homan  wrote:
> >
> > With my Mentor hat on, I'm entirely confident that Airflow is ready to
> > graduate.  The community broadly gets the Apache Way and operates
> > within it.  The community is healthy and engaged.  The last couple
> > releases went well, with no hitches whatsoever for the last one.
> >
> > The graduation process is mainly paperwork[1] and running VOTEs [2]
> > here and on the IPMC.  Last time I suggested this had to be done by
> > the Champion, which wasn't correct - anyone from the PPMC can do so.
> > I may have some free cycles over the next few weeks, so I'll take a
> > look at the check list to see what we can get out of the way, but any
> > of the PPMCers can also take items.  The IPMC also likes it if
> > Podlings go through and grade themselves on the Maturity Model[2], if
> > someone wants to do that.
> >
> > -Jakob
> >
> > [1] http://incubator.apache.org/projects/airflow.html
> > [2]
> http://mail-archives.apache.org/mod_mbox/airflow-dev/201809.mbox/%3CCAMdzn8vNbKQr2FF8WcJydFj16q0bgn0B42jfu5h28RS-ZQ=w...@mail.gmail.com%3E
> > [3]
> https://community.apache.org/apache-way/apache-project-maturity-model.html
> >> On Mon, Nov 26, 2018 at 1:10 PM Stefan Seelmann <
> m...@stefan-seelmann.de> wrote:
> >>
> >> I agree that Apache Airflow should graduate.
> >>
> >> I'm only involved since beginning of this year, but the project did two
> >> releases during that time, once TLP releasing becomes easier :)
> >>
> >> Regarding QU30 you may consider to use the ASF wide security mailing
> >> list [3] and process [4].
> >>
> >> Kind Regards,
> >> Stefan
> >>
> >> [3] https://www.apache.org/security/
> >> [4] https://www.apache.org/security/committers.html
> >>
> >>
> >>> On 11/26/18 8:46 PM, Bolke de Bruin wrote:
> >>> Ping!
> >>>
> >>> Sent from my iPhone
> >>>
>  On 24 Nov 2018, at 12:57, Bolke de Bruin  wrote:
> 
>  Hi All,
> 
>  With the Apache Airflow community healthy and growing, I think now
> would be a good time to
>  discuss where we stand regarding to graduation from the Incubator,
> and what requirements remains.
> 
>  Apache Airflow entered incubation around 2 years ago, since then, the
> Airflow community learned
>  a lot about how to do things in Apache ways. Now we are a very
> helpful and engaged community,
>  ready to help on all questions from the Airflow community. We
> delivered multiple releases that have
>  been increasing in quality ever since, now we can do self-driving
> releases in good cadence.
> 
>  The community is growing, new committers and PPMC members keep
> joining. We addressed almost all
>  the maturity issues stipulated by Apache Project Maturity Model [1].
> So final requirements remain, but
>  those just need a final nudge. Committers and contributors are
> invited to verify the list and pick up the last
>  bits (QU30, CO50). Finally (yahoo!) all the License and IP issues we
> can see got resolved.
> 
>  Base on those, I believes it's time for us to graduate to TLP. [2]
> Any thoughts?
>  And welcome advice from Airflow Mentors?
> 
>  Thanks,
> 
>  [1]
> https://cwiki.apache.org/confluence/display/AIRFLOW/Maturity+Evaluation
>  [2]
> https://incubator.apache.org/guides/graduation.html#graduating_to_a_top_level_project
> Regards,
> >>
>

Re: Airflow 1.10.1b1 release available - PLEASE TEST

[Hitesh Shah]

Hello Ash

For someone who is not familiar with the beta notation or folks who do not
check whether it is signed or not, could this be confused as an apache
release? Also, is there a plan to clean up/delete this version within a
finite time window once the official release voting is kicked off?

thanks
Hitesh


On Fri, Nov 9, 2018 at 5:52 AM Naik Kaxil  wrote:

> Good work Ash. Appreciate the clean and categorised Change Log.
>
> Regards,
> Kaxil
>
> On 09/11/2018, 13:45, "Ash Berlin-Taylor"  wrote:
>
> Hi Everyone,
>
> I've just released a beta version of 1.10.1! Please could you test
> this and report back any problems you notice, and also report back if you
> tried it and it works fine. As this is the first time I've released Airflow
> there is possible that there are packaging mistakes too. I'm not calling
> for a vote just yet, but I will give this a few days until I start making
> release candidates and calling for a formal vote, probably on Monday or
> Tuesday.
>
> In order to distinguish it from an actual (apache) release it is:
>
> 1. Marked as beta (python package managers do not install beta
> versions by default - PEP 440)
> 2. It is not signed
> 3. It is not at an official apache distribution location
>
> It can be installed with SLUGIFY_USES_TEXT_UNIDECODE=yes pip install
> 'apache-airflow==1.10.1b1'
>
> (Don't worry, without asking for `--pre` or specifying the version
> `pip install apache-airflow` will still get 1.10.0)
>
> Thanks,
> Ash
>
>
>
>
>
> Included below is the changelog of this release:
>
> New features:
>
> [AIRFLOW-2524] Airflow integration with AWS Sagemaker
> [AIRFLOW-2657] Add ability to delete DAG from web ui
> [AIRFLOW-2780] Adds IMAP Hook to interact with a mail server
> [AIRFLOW-2794] Add delete support for Azure blob
> [AIRFLOW-2912] Add operators for Google Cloud Functions
> [AIRFLOW-2974] Add Start/Restart/Terminate methods Databricks Hook
> [AIRFLOW-2989] No Parameter to change bootDiskType for
> DataprocClusterCreateOperator
> [AIRFLOW-3078] Basic operators for Google Compute Engine
> [AIRFLOW-3147] Update Flask-AppBuilder version
> [AIRFLOW-3231] Basic operators for Google Cloud SQL (deploy / patch /
> delete)
> [AIRFLOW-3276] Google Cloud SQL database create / patch / delete
> operators
>
> Improvements:
>
> [AIRFLOW-393] Add progress callbacks for FTP downloads
> [AIRFLOW-520] Show Airflow version on web page
> [AIRFLOW-843] Excpetions now available in context durint
> on_failure_callback
> [AIRFLOW-2476] Update tabulate dependency to v0.8.2
> [AIRFLOW-2592] Bump Bleach dependency
> [AIRFLOW-2622] Add "confirm=False" option to SFTPOperator
> [AIRFLOW-2662] support affinity & nodeSelector policies for kubernetes
> executor/operator
> [AIRFLOW-2709] Improve error handling in Databricks hook
> [AIRFLOW-2763] No precheck mechanism in place during worker
> initialisation for the connection to metadata database
> [AIRFLOW-2789] Add ability to create single node cluster to
> DataprocClusterCreateOperator
> [AIRFLOW-2797] Add ability to create Google Dataproc cluster with
> custom image
> [AIRFLOW-2854] kubernetes_pod_operator add more configuration items
> [AIRFLOW-2855] Need to Check Validity of Cron Expression When Process
> DAG File/Zip File
> [AIRFLOW-2904] Clean an unnecessary line in
> airflow/executors/celery_executor.py
> [AIRFLOW-2921] A trivial incorrectness in CeleryExecutor()
> [AIRFLOW-2922] Potential deal-lock bug in CeleryExecutor()
> [AIRFLOW-2932] GoogleCloudStorageHook - allow compression of file
> [AIRFLOW-2949] Syntax Highlight for Single Quote
> [AIRFLOW-2951] dag_run end_date Null after a dag is finished
> [AIRFLOW-2956] Kubernetes tolerations for pod operator
> [AIRFLOW-2997] Support for clustered tables in Bigquery hooks/operators
> [AIRFLOW-3006] Fix rrror when schedule_interval="None"
> [AIRFLOW-3008] Move Kubernetes related example DAGs to
> contribe/example_dags
> [AIRFLOW-3025] Allow to specify dns and dns-search parameters for
> DockerOperator
> [AIRFLOW-3067] (www_rbac) Flask flash messages are not displayed
> properly (no background color)
> [AIRFLOW-3069] Decode output of S3 file transform operator
> [AIRFLOW-3090] INFO logs are too verbose
> [AIRFLOW-3103] Update Flask-Login
> [AIRFLOW-3112] Align SFTP hook with SSH hook
> [AIRFLOW-3119] Enable loglevel on celery worker and inherit from
> airflow.cfg
> [AIRFLOW-3137] Make ProxyFix middleware optional
> [AIRFLOW-3173] Add _cmd options for more password config options
> [AIRFLOW-3177] Change scheduler_heartbeat metric from gauge to counter
> [AIRFLOW-3195] Druid Hook: Log ingestion spec and task id
> [AIRFLOW-3197] EMR Hook is missing some parameters to valid on the AWS
> API
> [AIRFLOW-3232] Make documentation for GCF Functions operator more
> 

Re: Airflow: Apache Graduation

[Hitesh Shah]

The champion need not be the one to do all of this. Anyone from the PPMC
can help with the task list. One point to add to Jakob's list (given the
general popularity of Airflow) is the press release to announce Airflow as
a TLP as soon as the board approves the resolution. This will require some
coordination with Sally and help from the PPMC depending on the contents of
the press release.

thanks
HItesh

On Tue, Sep 25, 2018 at 9:17 AM Chris Riccomini 
wrote:

> Hey all,
>
> If I'm being honest with myself, I don't believe I'll have enough time to
> do this adequately. Sorry about this, but I'm having to focus on a number
> of different areas at work these days, and it's leaving a bit less time for
> Airflow. I do plan to keep involved, but I don't think steering Airflow
> through graduation is the wisest thing for me to be doing at the moment.
>
> Cheers,
> Chris
>
> On Fri, Sep 21, 2018 at 3:04 PM Bolke de Bruin  wrote:
>
> > I love champions 
> >
> > Op vr 21 sep. 2018 14:01 schreef Jakob Homan :
> >
> > > The big list that *has* to be filled out and correct is here:
> > > http://incubator.apache.org/projects/airflow.html
> > >
> > > This is usually done by the Champion, Chris Riccomini.  Chris  - are
> > > you still interested to do this?
> > >
> > > After that and the maturity questionnaire are in good shape, it's:
> > > 1) Graduation DISCUSS by the community.  This thread can cover that,
> > > although an explicit DISCUSS would help too.  Example:
> > >
> > >
> >
> https://mail-archives.apache.org/mod_mbox/samza-dev/201412.mbox/%3CCADiKvVtBOPaji22Nd3927R6Tj%2BmWpxnHH%2Bk%3DyXCQEMGf_CnAqw%40mail.gmail.com%3E
> > > 2) Graduation VOTE by the community on a resolution for the new top
> > > level project (TLP). Example:
> > >
> > >
> >
> https://mail-archives.apache.org/mod_mbox/samza-dev/201412.mbox/%3CCADiKvVt_LK9%2BBXC8eR_gb3ALOU2KFJqDDyZOLao7rm0XjuZmcg%40mail.gmail.com%3E
> > > - most of the text is boiler plate, but be sure not leave anything in
> > > there that shouldn't be (Jakob said with experience...)
> > > 3) Exact same text is VOTEd on by the IPMC
> > > 4) Champion adds the resolution to the Board's next monthly agenda,
> > > where the Board approves it.  Project is officially TLP once that is
> > > done.
> > > 5) Champion cleans up incubator resources, creates new TLP resources.
> > > Example: https://jira.apache.org/jira/browse/INFRA-10639
> > >
> > > -Jakob
> > >
> > >
> > > On 21 September 2018 at 13:49, Bolke de Bruin 
> wrote:
> > > > There is a maturity check that we can do. Don't have the link ready
> > now.
> > > > But can look it up later.
> > > >
> > > > To me two things need to be more mature
> > > >
> > > > 1. Handling of security issues
> > > > 2. Licensing.
> > > >  2a having ppl aware that with new dependencies updates to
> dependencies
> > > the
> > > > respective licenses need to be updated. Too often I need to fix some
> > > issues
> > > > at release time.
> > > >  2b versions need to be added to the licenses (version of the
> software)
> > > >
> > > > B.
> > > >
> > > > Op vr 21 sep. 2018 13:00 schreef Ash Berlin-Taylor :
> > > >
> > > >> Your guess is as good as mine on what is involved in graduation. I
> > think
> > > >> that we sorted the Licensing issues in 1.10.0 (even if the way we
> > > sorted it
> > > >> was a little annoying - having to specify the environment variable
> at
> > > `pip
> > > >> install` time is a littlle bit un-pythonic, but I wasn't thinking of
> > > fixing
> > > >> that in 1.10.1)
> > > >>
> > > >> Some of the steps and requirements are listed here
> > > >> graduating_to_a_top_level_project <
> > > >> https://incubator.apache.org/guides/graduation.html>, but in
> summary
> > > from
> > > >> a quick read of it I think the process is:
> > > >>
> > > >> - Collect some information, put it on
> > > >> http://incubator.apache.org/projects/airflow.html <
> > > >> http://incubator.apache.org/projects/airflow.html> for the IPMC
> > > >> - We as a community hold a vote on if we think we're ready to
> graduate
> > > >> - IPMC vote on it too
> > > >> - propose motion for (monthly) Apache board meeting
> > > >>
> > > >> There might be a few more steps involves, such as drafting a Charter
> > (we
> > > >> would probably start with a "stock" Apache one)
> > > >>
> > > >> -ash
> > > >>
> > > >> > On 20 Sep 2018, at 18:22, Maxime Beauchemin <
> > > maximebeauche...@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> > Yeah let's make it happen! I'm happy to set some time aside to
> help
> > > with
> > > >> > the final push.
> > > >> >
> > > >> > Max
> > > >> >
> > > >> > On Thu, Sep 20, 2018 at 9:53 AM Sid Anand 
> > wrote:
> > > >> >
> > > >> >> Folks! (specifically Bolke, Fokko, Ash)
> > > >> >> What's needed to graduate from Apache?
> > > >> >>
> > > >> >> Can we make 1.10.1 be about meeting our licensing needs to allow
> us
> > > to
> > > >> >> graduate?
> > > >> >>
> > > >> >> -s
> > > >> >>
> > > >>
> > > >>
> > >
> >
>

Airflow talks at Adobe

[Hitesh Shah]

Hi folks

[Putting my Adobe employee hat on]

We, at Adobe, have a bunch of folks who are trying out Airflow and a lot of
others who are interested in using it.

Would any committer/PMC member would willing to come to Adobe (downtown San
Jose, CA) to give an overview talk on Airflow and help answer more in-depth
questions for folks who are interested in using Airflow at scale?

Additionally, is there someone from the operations side who
manages/monitors Airflow at scale and would be willing to do an Airflow:
from the Ops trenches kind of talk in addition to the overview?

If yes, please drop me a direct note at hitesh at apache.org

thanks
Hitesh

Re: [VOTE] Release Airflow 1.8.2 based on Airflow 1.8.2 RC3

[Hitesh Shah]

Another clarification - if the plan is to publish to the binaries after the
vote, I would recommend having the binaries be generated and made available
as part of the vote to verify that the contents/license/notice files are
correct.

thanks
-- HItesh

On Thu, Aug 3, 2017 at 1:37 PM, Hitesh Shah <hit...@apache.org> wrote:

> Thanks for the clarification, Ash.
>
> In that case, I am assuming that a release push should be okay as long as
> the package description clearly indicates "incubating" and the version is
> "1.8.2" (no rc suffix though - only a voted upon and verified release
> should be published). I would recommend calling this out in the IPMC vote
> to get general feedback if this has not been already addressed in other
> threads.
>
> thanks
> -- Hitesh
>
> On Thu, Aug 3, 2017 at 12:09 PM, Ash Berlin-Taylor <
> ash_airflowl...@firemirror.com> wrote:
>
>> pip can be very picky at times the version number of things it installs,
>> specifically when installing from wheels (which are pre-packaged, and
>> recommend because they are much faster to install.)
>>
>> A distribution version number should follow PEP-0440
>> https://www.python.org/dev/peps/pep-0440/ <https://www.python.org/dev/pe
>> ps/pep-0440/> which says:
>>
>> > # Public version identifiers
>> >
>> > The canonical public version identifiers MUST comply with the following
>> scheme:
>> >
>> >     [N!]N(.N)*[{a|b|rc}N][.postN][.devN]
>>
>> So according to PEP440 "incubating" MUST (in the RFC sence) not appear in
>> the version number.
>>
>>
>>
>> > On 3 Aug 2017, at 19:56, Hitesh Shah <hit...@apache.org> wrote:
>> >
>> > -1
>> >
>> > Verified signatures and checksums.
>> > Verified Disclaimer.
>> > Ran rat-check
>> > Verified License for source. Quick question on the bundled images - are
>> > these covered by the License file?
>> > Version number incorrect and does not contain "incubating".
>> >
>> > The main (albeit a very minor change) issue is that the version in
>> > version.py is "1.8.2rc2". If there is a plan to publish the binary
>> > artifacts, the version should be "1.8.2-incubating". Changing this means
>> > that the tarball needs to be modified and would no longer be the
>> artifact
>> > that was voted upon.
>> >
>> > thanks
>> > -- Hitesh
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Wed, Aug 2, 2017 at 1:17 PM, Chris Riccomini <criccom...@apache.org>
>> > wrote:
>> >
>> >> Verified asc, sha, and md5. Thanks, Max!
>> >>
>> >> +1 (binding)
>> >>
>> >> On Wed, Aug 2, 2017 at 12:56 PM, Maxime Beauchemin <
>> >> maximebeauche...@gmail.com> wrote:
>> >>
>> >>> It's the same, only the bundling is different. In this case we're
>> >> bundling
>> >>> the source with and INSTALL file as opposed to the SDIST binaries.
>> >>>
>> >>> If it goes well I don't mind doing 1.9.0 soon after.
>> >>>
>> >>> Max
>> >>>
>> >>> On Wed, Aug 2, 2017 at 12:37 PM, Bolke de Bruin <bdbr...@gmail.com>
>> >> wrote:
>> >>>
>> >>>> Assuming same as rc2 which we are running for over a month now.
>> >>>>
>> >>>> +1 (binding)
>> >>>>
>> >>>> Bolke
>> >>>>
>> >>>> Sent from my iPhone
>> >>>>
>> >>>>> On 1 Aug 2017, at 23:52, Maxime Beauchemin <
>> >> maximebeauche...@gmail.com
>> >>>>
>> >>>> wrote:
>> >>>>>
>> >>>>> 1.8.2 RC3 is baked and available at:
>> >>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow, public
>> >> keys
>> >>>>> are available
>> >>>>> at https://dist.apache.org/repos/dist/release/incubator/airflow.
>> >>>>>
>> >>>>> This is a source release that comes with INSTALL instructions.
>> >>>>>
>> >>>>> 1.8.2 RC3 is build upon 1.8.1 with the commits listed bellow on top
>> >> of
>> >>>> it.
>> >>>>> I added the JIRAs that were identified blockers and targeted 1.8.2.
>>

Re: [VOTE] Release Airflow 1.8.2 based on Airflow 1.8.2 RC3

[Hitesh Shah]

Thanks for the clarification, Ash.

In that case, I am assuming that a release push should be okay as long as
the package description clearly indicates "incubating" and the version is
"1.8.2" (no rc suffix though - only a voted upon and verified release
should be published). I would recommend calling this out in the IPMC vote
to get general feedback if this has not been already addressed in other
threads.

thanks
-- Hitesh

On Thu, Aug 3, 2017 at 12:09 PM, Ash Berlin-Taylor <
ash_airflowl...@firemirror.com> wrote:

> pip can be very picky at times the version number of things it installs,
> specifically when installing from wheels (which are pre-packaged, and
> recommend because they are much faster to install.)
>
> A distribution version number should follow PEP-0440
> https://www.python.org/dev/peps/pep-0440/ <https://www.python.org/dev/
> peps/pep-0440/> which says:
>
> > # Public version identifiers
> >
> > The canonical public version identifiers MUST comply with the following
> scheme:
> >
> > [N!]N(.N)*[{a|b|rc}N][.postN][.devN]
>
> So according to PEP440 "incubating" MUST (in the RFC sence) not appear in
> the version number.
>
>
>
> > On 3 Aug 2017, at 19:56, Hitesh Shah <hit...@apache.org> wrote:
> >
> > -1
> >
> > Verified signatures and checksums.
> > Verified Disclaimer.
> > Ran rat-check
> > Verified License for source. Quick question on the bundled images - are
> > these covered by the License file?
> > Version number incorrect and does not contain "incubating".
> >
> > The main (albeit a very minor change) issue is that the version in
> > version.py is "1.8.2rc2". If there is a plan to publish the binary
> > artifacts, the version should be "1.8.2-incubating". Changing this means
> > that the tarball needs to be modified and would no longer be the artifact
> > that was voted upon.
> >
> > thanks
> > -- Hitesh
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Wed, Aug 2, 2017 at 1:17 PM, Chris Riccomini <criccom...@apache.org>
> > wrote:
> >
> >> Verified asc, sha, and md5. Thanks, Max!
> >>
> >> +1 (binding)
> >>
> >> On Wed, Aug 2, 2017 at 12:56 PM, Maxime Beauchemin <
> >> maximebeauche...@gmail.com> wrote:
> >>
> >>> It's the same, only the bundling is different. In this case we're
> >> bundling
> >>> the source with and INSTALL file as opposed to the SDIST binaries.
> >>>
> >>> If it goes well I don't mind doing 1.9.0 soon after.
> >>>
> >>> Max
> >>>
> >>> On Wed, Aug 2, 2017 at 12:37 PM, Bolke de Bruin <bdbr...@gmail.com>
> >> wrote:
> >>>
> >>>> Assuming same as rc2 which we are running for over a month now.
> >>>>
> >>>> +1 (binding)
> >>>>
> >>>> Bolke
> >>>>
> >>>> Sent from my iPhone
> >>>>
> >>>>> On 1 Aug 2017, at 23:52, Maxime Beauchemin <
> >> maximebeauche...@gmail.com
> >>>>
> >>>> wrote:
> >>>>>
> >>>>> 1.8.2 RC3 is baked and available at:
> >>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow, public
> >> keys
> >>>>> are available
> >>>>> at https://dist.apache.org/repos/dist/release/incubator/airflow.
> >>>>>
> >>>>> This is a source release that comes with INSTALL instructions.
> >>>>>
> >>>>> 1.8.2 RC3 is build upon 1.8.1 with the commits listed bellow on top
> >> of
> >>>> it.
> >>>>> I added the JIRAs that were identified blockers and targeted 1.8.2. I
> >>>>> attempted to bring in all of the JIRAs that targeted 1.8.2 but bailed
> >>> on
> >>>>> the ones that were generating merge conflicts. I also added all of
> >> the
> >>>>> JIRAs that we've been running in production at Airbnb.
> >>>>>
> >>>>> Issues fixed:
> >>>>> 9a53e66 [AIRFLOW-809][AIRFLOW-1] Use __eq__ ColumnOperator When
> >> Testing
> >>>>> Booleans
> >>>>> 333e0b3 [AIRFLOW-1296] Propagate SKIPPED to all downstream tasks
> >>>>> 93825d5 [AIRFLOW-XXX] Re-enable caching for hadoop components
> >>>>> 33a9dcb [AIRFLOW-XXX] Pin Hive and Hadoop to a specific version an

Re: [VOTE] Release Airflow 1.8.2 based on Airflow 1.8.2 RC3

[Hitesh Shah]

-1

Verified signatures and checksums.
Verified Disclaimer.
Ran rat-check
Verified License for source. Quick question on the bundled images - are
these covered by the License file?
Version number incorrect and does not contain "incubating".

The main (albeit a very minor change) issue is that the version in
version.py is "1.8.2rc2". If there is a plan to publish the binary
artifacts, the version should be "1.8.2-incubating". Changing this means
that the tarball needs to be modified and would no longer be the artifact
that was voted upon.

thanks
-- Hitesh









On Wed, Aug 2, 2017 at 1:17 PM, Chris Riccomini 
wrote:

> Verified asc, sha, and md5. Thanks, Max!
>
> +1 (binding)
>
> On Wed, Aug 2, 2017 at 12:56 PM, Maxime Beauchemin <
> maximebeauche...@gmail.com> wrote:
>
> > It's the same, only the bundling is different. In this case we're
> bundling
> > the source with and INSTALL file as opposed to the SDIST binaries.
> >
> > If it goes well I don't mind doing 1.9.0 soon after.
> >
> > Max
> >
> > On Wed, Aug 2, 2017 at 12:37 PM, Bolke de Bruin 
> wrote:
> >
> > > Assuming same as rc2 which we are running for over a month now.
> > >
> > > +1 (binding)
> > >
> > > Bolke
> > >
> > > Sent from my iPhone
> > >
> > > > On 1 Aug 2017, at 23:52, Maxime Beauchemin <
> maximebeauche...@gmail.com
> > >
> > > wrote:
> > > >
> > > > 1.8.2 RC3 is baked and available at:
> > > > https://dist.apache.org/repos/dist/dev/incubator/airflow, public
> keys
> > > > are available
> > > > at https://dist.apache.org/repos/dist/release/incubator/airflow.
> > > >
> > > > This is a source release that comes with INSTALL instructions.
> > > >
> > > > 1.8.2 RC3 is build upon 1.8.1 with the commits listed bellow on top
> of
> > > it.
> > > > I added the JIRAs that were identified blockers and targeted 1.8.2. I
> > > > attempted to bring in all of the JIRAs that targeted 1.8.2 but bailed
> > on
> > > > the ones that were generating merge conflicts. I also added all of
> the
> > > > JIRAs that we've been running in production at Airbnb.
> > > >
> > > > Issues fixed:
> > > > 9a53e66 [AIRFLOW-809][AIRFLOW-1] Use __eq__ ColumnOperator When
> Testing
> > > > Booleans
> > > > 333e0b3 [AIRFLOW-1296] Propagate SKIPPED to all downstream tasks
> > > > 93825d5 [AIRFLOW-XXX] Re-enable caching for hadoop components
> > > > 33a9dcb [AIRFLOW-XXX] Pin Hive and Hadoop to a specific version and
> > > create
> > > > writable warehouse dir
> > > > 7cff6cd [AIRFLOW-1308] Disable nanny usage for Dask
> > > > 570b2ed [AIRFLOW-1294] Backfills can loose tasks to execute
> > > > 3f48d48 [AIRFLOW-1291] Update NOTICE and LICENSE files to match ASF
> > > > requirements
> > > > 69bd269 [AIRFLOW-1160] Update Spark parameters for Mesos
> > > > 9692510 [AIRFLOW 1149][AIRFLOW-1149] Allow for custom filters in
> Jinja2
> > > > templates
> > > > 6de5330 [AIRFLOW-1119] Fix unload query so headers are on first row[]
> > > > b4e9eb8 [AIRFLOW-1089] Add Spark application arguments
> > > > a4083f3 [AIRFLOW-1078] Fix latest_runs endpoint for old flask
> versions
> > > > 7a02841 [AIRFLOW-1074] Don't count queued tasks for concurrency
> limits
> > > > a2c18a5 [AIRFLOW-1064] Change default sort to job_id for
> > > > TaskInstanceModelView
> > > > d1c64ab [AIRFLOW-1038] Specify celery serialization options
> explicitly
> > > > b4ee88a [AIRFLOW-1036] Randomize exponential backoff
> > > > 9fca409 [AIRFLOW-993] Update date inference logic
> > > > 272c2f5 [AIRFLOW-1167] Support microseconds in FTPHook modification
> > time
> > > > c7c0b72 [AIRFLOW-1179] Fix Pandas 0.2x breaking Google BigQuery
> change
> > > > acd0166 [AIRFLOW-1263] Dynamic height for charts
> > > > 7f33f6e [AIRFLOW-1266] Increase width of gantt y axis
> > > > fc33c04 [AIRFLOW-1290] set docs author to 'Apache Airflow'
> > > > 2e9eee3 [AIRFLOW-1282] Fix known event column sorting
> > > > 2389a8a [AIRFLOW-1166] Speed up _change_state_for_tis_without_dagrun
> > > > bf966e6 [AIRFLOW-1192] Some enhancements to qubole_operator
> > > > 57d5bcd [AIRFLOW-1281] Sort variables by key field by default
> > > > 802fc15 [AIRFLOW-1244] Forbid creation of a pool with empty name
> > > > 1232b6a [AIRFLOW-1243] DAGs table has no default entries to show
> > > > b0ba3c9 [AIRFLOW-1227] Remove empty column on the Logs view
> > > > c406652 [AIRFLOW-1226] Remove empty column on the Jobs view
> > > > 51a83cc [AIRFLOW-1199] Fix create modal
> > > > cac7d4c [AIRFLOW-1200] Forbid creation of a variable with an empty
> key
> > > > 5f3ee52 [AIRFLOW-1186] Sort dag.get_task_instances by execution_date
> > > > f446c08 [AIRFLOW-1145] Fix closest_date_partition function with
> before
> > > set
> > > > to True If we're looking for the closest date before, we should take
> > the
> > > > latest date in the list of date before.
> > > > 93b8e96 [AIRFLOW-1180] Fix flask-wtf version for test_csrf_rejection
> > > > bb56805 [AIRFLOW-1170] DbApiHook insert_rows inserts parameters
> > > separately
> > > > 093b2f0 

Re: [VOTE] Release Airflow 1.8.2 based on Airflow 1.8.2 RC2

[Hitesh Shah]

To add, the main source tarball should have instructions to generate the
sdist and bdist versions. Additionally, as part of the release process if
the plan is to publish to pypi (after the IPMC vote succeeds), then the
appropriate bits also need to be verified/voted upon. There are not exactly
counted as the official release bits but they do need to be verified as
part of the voting process to ensure that the bits do indeed map to the
source release, license/notice files are correct, etc.

thanks
-- Hitesh


On Tue, Jul 18, 2017 at 12:01 AM, Bolke de Bruin  wrote:

> Thanks Hitesh. We discussed it with John Ament on the IPMC. Python has the
> notion of 3 types of distributions, “source”, “sdist”, “bdist”, contrary to
> Java that knows only two (source, bdist). We used to vote on “sdist”, which
> was deemed incorrect.
>
> So, Max, indeed we need to vote on a tar.gz that contains build
> instructions in INSTALL to get to “sdist”. The build instructions should
> also contain instruction how to run the license checks by Apache Rat. Most
> of the work probably goes in the build instructions and verifying they
> work, but it should not be much.
>
> Any other clarification required?
>
> Bolke
>
>

Re: [VOTE] Release Airflow 1.8.1 based on Airflow 1.8.1 RC0

[Hitesh Shah]

Hi Bolke,

I dont believe a non-release version would be ever published publicly
outside of dist.apache.org/dev/.

I am not sure what a good answer is. One thing I can think of is that given
that vote is really only for the source bits and as long as the source
tarball/release commit hash is not being modified after the vote completes
and the community (PMC) trusts the release manager to publish the correctly
modified convenience artifacts (binaries), it should be okay. But this is
something the community needs to agree/resolve together.

-- Hitesh


On Sun, Apr 23, 2017 at 12:17 AM, Bolke de Bruin <bdbr...@gmail.com> wrote:

>
>
> Sent from my iPhone
>
> > On 23 Apr 2017, at 03:46, Hitesh Shah <hit...@apache.org> wrote:
> >
> > On Fri, Apr 21, 2017 at 8:19 AM, Chris Riccomini <criccom...@apache.org>
> > wrote:
> >
> >>
> >>> Version in pkg-info has an rc0 notation. It should just be
> >> 1.8.1-incubating.
> >>
> >> This is a bit tricky to do with Python builds. I don't really want to
> keep
> >> building RCs with the exact same version number. We bake these RCs in
> real
> >> environments, so we need to version them with something that
> distinguishes
> >> one from another. Once we set the version, that propagates into the
> >> pkg-info. The plan is to rebuild the final RC that passes without the rc
> >> notation, so the release doesn't contain it.
> >>
> >>
> > I understand the rationale but this means that there is a potential
> > difference in what is being voted upon and what is eventually being
> > published as a release.
> >
> > thanks
> > -- Hitesh
>
> Hi Hitesh,
>
> This is a chicken and egg problem. If we put a non release 1.8.1 online
> users will download  and install it. An update to this package will not
> trigger an upgrade on the user's side and it is hard to recognize (one will
> need to compare signature). It puts them at risk.
>
> Do you know how other python projects solved this? I will reach out to the
> libcloud guys and ask them how they did it (also python).
>
> Bolke

Re: [VOTE] Release Airflow 1.8.1 based on Airflow 1.8.1 RC0

[Hitesh Shah]

On Fri, Apr 21, 2017 at 8:19 AM, Chris Riccomini 
wrote:

>
> > Version in pkg-info has an rc0 notation. It should just be
> 1.8.1-incubating.
>
> This is a bit tricky to do with Python builds. I don't really want to keep
> building RCs with the exact same version number. We bake these RCs in real
> environments, so we need to version them with something that distinguishes
> one from another. Once we set the version, that propagates into the
> pkg-info. The plan is to rebuild the final RC that passes without the rc
> notation, so the release doesn't contain it.
>
>
I understand the rationale but this means that there is a potential
difference in what is being voted upon and what is eventually being
published as a release.

thanks
-- Hitesh

Re: [VOTE] Release Airflow 1.8.1 based on Airflow 1.8.1 RC0

[Hitesh Shah]

-1.

Not sure if these have been called out earlier.

For all the bundled files with different licenses (MIT, BSD, etc), the full
texts of these licenses should be in the source tarball preferably at the
end of the LICENSE file.
webgl-2d needs to be called out as MIT license.
Version in pkg-info has an rc0 notation. It should just be
1.8.1-incubating.
A bunch of files under apache_airflow.egg-info/ and scripts/systemd/ need a
license header
Likewise for airflow/www/templates/airflow/variables/README.md

Nice to have:
Fix the top-level dir in the tarball to be
"apache-airflow-1.8.1-incubating" instead of
"apache-airflow-1.8.1rc0+apache.incubating"

For all the other binary files (images, gifs), is there source provenance
for all of them and that all of them are covered by the licenses in the
LICENSE file?

Last point - are all the entries in the NOTICE file required or do they
just need to be in the LICENSE file? Any additions to the NOTICE have
downstream repercussions as they need to be propagated down by any other
project using airflow.

thanks
-- Hitesh



On Mon, Apr 17, 2017 at 11:24 AM, Chris Riccomini 
wrote:

> Dear All,
>
> I have been able to make the Airflow 1.8.1 RC0 available at:
> https://dist.apache.org/repos/dist/dev/incubator/airflow, public keys are
> available at https://dist.apache.org/repos/dist/release/incubator/airflow.
>
> Issues fixed:
>
> [AIRFLOW-1062] DagRun#find returns wrong result if external_trigg
> [AIRFLOW-1054] Fix broken import on test_dag
> [AIRFLOW-1050] Retries ignored - regression
> [AIRFLOW-1033] TypeError: can't compare datetime.datetime to None
> [AIRFLOW-1030] HttpHook error when creating HttpSensor
> [AIRFLOW-1017] get_task_instance should return None instead of th
> [AIRFLOW-1011] Fix bug in BackfillJob._execute() for SubDAGs
> [AIRFLOW-1001] Landing Time shows "unsupported operand type(s) fo
> [AIRFLOW-1000] Rebrand to Apache Airflow instead of Airflow
> [AIRFLOW-989] Clear Task Regression
> [AIRFLOW-974] airflow.util.file mkdir has a race condition
> [AIRFLOW-906] Update Code icon from lightning bolt to file
> [AIRFLOW-858] Configurable database name for DB operators
> [AIRFLOW-853] ssh_execute_operator.py stdout decode default to A
> [AIRFLOW-832] Fix debug server
> [AIRFLOW-817] Trigger dag fails when using CLI + API
> [AIRFLOW-816] Make sure to pull nvd3 from local resources
> [AIRFLOW-815] Add previous/next execution dates to available def
> [AIRFLOW-813] Fix unterminated unit tests in tests.job (tests/jo
> [AIRFLOW-812] Scheduler job terminates when there is no dag file
> [AIRFLOW-806] UI should properly ignore DAG doc when it is None
> [AIRFLOW-794] Consistent access to DAGS_FOLDER and SQL_ALCHEMY_C
> [AIRFLOW-785] ImportError if cgroupspy is not installed
> [AIRFLOW-784] Cannot install with funcsigs > 1.0.0
> [AIRFLOW-780] The UI no longer shows broken DAGs
> [AIRFLOW-777] dag_is_running is initlialized to True instead of
> [AIRFLOW-719] Skipped operations make DAG finish prematurely
> [AIRFLOW-694] Empty env vars do not overwrite non-empty config v
> [AIRFLOW-139] Executing VACUUM with PostgresOperator
> [AIRFLOW-111] DAG concurrency is not honored
> [AIRFLOW-88] Improve clarity Travis CI reports
>
> I would like to raise a VOTE for releasing 1.8.1 based on release candidate
> 0, i.e. just renaming release candidate 0 to 1.8.1 release.
>
> Please respond to this email by:
>
> +1,0,-1 with *binding* if you are a PMC member or *non-binding* if you are
> not.
>
> Vote will run for 72 hours (ends this Thursday).
>
> Thanks!
> Chris
>
> My VOTE: +1 (binding)
>

Re: PTAL: Airflow 2017 April Podling Report

[Hitesh Shah]

Can someone please ensure that
http://incubator.apache.org/projects/airflow.html (annoucements section) is
updated with information on all the releases, new committers/PPMC members,
etc?

Additionally, if the community is considering that they are ready to
graduate, it would be a good idea to look at
https://community.apache.org/apache-way/apache-project-maturity-model.html
and see where the project stands in this regard.

thanks
-- Hitesh

On Wed, Apr 5, 2017 at 4:59 PM, siddharth anand <san...@apache.org> wrote:

> Just reviewed the report. Looks great! It is ready for sign-off from our
> mentors.
> -s
>
> On Wed, Apr 5, 2017 at 3:49 PM, Arthur Wiedmer <arthur.wied...@gmail.com>
> wrote:
>
> > I added the following to the report, as it seemed like it was a required
> > question :
> >
> > How would you assess the podling's maturity?Please feel free to add
> > your own commentary.  [ ] Initial setup  [ ] Working towards first
> > release  [ ] Community building  [X] Nearing graduation  [ ] Other:The
> > Airflow community continues to grow, and we have successfully created
> > our first Apache release. We want to continue on this momentum and
> > create another release to solidify our  process and tools around it,
> > but we feel we are nearing graduation. We are open to feedback and
> > guidance to make sure we can do so.
> >
> > Does this look OK ? I only have until tonight to change it.
> > To view the report : https://wiki.apache.org/incubator/April2017
> >
> > Best,
> > Arthur
> >
> >
> >
> > On Wed, Apr 5, 2017 at 11:57 AM, Gurer Kiratli <
> > gurer.kira...@airbnb.com.invalid> wrote:
> >
> > > Hi folks,
> > >
> > > Here is the draft of the podling report. Please take a look and
> comment.
> > If
> > > it looks good one of the committers have to post this on this on the
> wiki
> > > today!
> > >
> > > Cheers,
> > >
> > > Gurer
> > >
> > >
> > >
> > > >>
> > >
> > > Airflow
> > >
> > > Airflow is a workflow automation and scheduling system that can be used
> > to
> > > author and manage data pipelines.
> > >
> > > Airflow has been incubating since 2016-03-31.
> > >
> > > Three most important issues to address in the move towards graduation:
> > >
> > >   1. We will have the 1.8.1 release soon, then we are looking to
> > graduate.
> > >   2.
> > >   3.
> > >
> > > Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
> > > aware of?
> > >
> > >   None
> > >
> > > How has the community developed since the last report?
> > >
> > >   1. We had our first official release. 1.8.0 on March 19th 2017.
> > >   2. We elected 1 new PPMC Member/Committer: Alex Guziel (a.k.a
> saguziel)
> > >   3. Since our last podling report 3 months ago (i.e. between Jan 1 and
> > Mar
> > >  31, inclusive), we grew our contributors from 224 to 256
> > >   4. Since our last podling report 3 months ago (i.e. between Jan 1 and
> > Mar
> > >  31, inclusive), we resolved 216 pull requests (currently at 1479
> > > closed
> > >  PRs)
> > >   5. Two meet-ups, one in New York, NY hosted by Blue Apron and one in
> > San
> > > Jose, CA hosted by PayPal were held by the community.
> > >   6. Since being accepted into the incubator, the number of companies
> > >  officially using Apache Airflow has risen from 30 to 83.
> > >
> > > How has the project developed since the last report?
> > >
> > >   See above
> > >
> > > Date of last release:
> > >
> > >   March 19th, 2017
> > >
> > > When were the last committers or PPMC members elected?
> > >
> > >   As mentioned on
> > >
> > > https://cwiki.apache.org/confluence/display/AIRFLOW/
> > > Announcements#Announcements-Mar14,2017
> > >   Alex Guziel joined the Apache Airflow PPMC/Committer group.
> > >
> > > Signed-off-by:
> > >
> > >   [](airflow) Chris Nauroth
> > >   [](airflow) Hitesh Shah
> > >   [ ](airflow) Jakob Homan
> > >
> >
>

Re: Astronomer.io Airflow blog post

[Hitesh Shah]

Great article. Could someone from the PMC reach out to the author to fix the 
article to follow the guidelines at http://www.apache.org/foundation/marks/?  
(the most important of which is that the first use of Airflow should be 
qualified with “Apache” and likewise for a whole bunch of other ASF projects 
referenced )

thanks 
— Hitesh


> On Nov 2, 2016, at 3:04 PM, Gerard Toonstra  wrote:
> 
> Thanks!
> 
> On Wed, Nov 2, 2016 at 10:45 PM, Maxime Beauchemin <
> maximebeauche...@gmail.com> wrote:
> 
>> in the meantime I added a link to your docs here:
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Links
>> 
>> On Wed, Nov 2, 2016 at 12:33 PM, siddharth anand 
>> wrote:
>> 
>>> Gerard,
>>> Please sign up for a CWiki 
>>> account and reply to this email with your user name. I just searched
>>> for "Gerard
>>> Toonstra" and didn't find a user on CWiki. Once you register a login, I
>>> will grant your account admin perms. You are free to arrange the current
>>> page of links and to add anything missing as you see fit.
>>> 
>>> -s
>>> 
>>> On Wed, Nov 2, 2016 at 12:27 PM, Gerard Toonstra 
>>> wrote:
>>> 
 talking about documentation :)
 
 I made the changes re: comments by Boris:
 
 https://gtoonstra.github.io/etl-with-airflow/
 
 First level of review:
 - does the organization of tasks in DAGs make sense,
 - would you split it up tasks like this or would you put everything
 together?
 - task dependencies : "depends_on_past"  recommended, overall DAG
 configuration
 - better ways to organize a DAG to reuse it for "daily" import vs. a
>> more
 historical "monthly" or "weekly"  import?
   (what's best way to do historical reloads? ).
 
 
 All source is currently at:
 
 https://github.com/gtoonstra/etl-with-airflow
 
 Happy ot move it elsewhere to make it more of a community project, but
>> if
 there's lack of time, I'd like to ask you
 to add the site (top link) to the wiki as well, then I'll keep charge
>> of
>>> it
 until we maybe decide otherwise in the future.
 
 Rgds,
 
 Gerard
 
 On Wed, Nov 2, 2016 at 7:43 PM, Maxime Beauchemin <
 maximebeauche...@gmail.com> wrote:
 
> Here:
> https://github.com/apache/incubator-airflow/pull/1863
> 
> 
> On Wed, Nov 2, 2016 at 10:51 AM, siddharth anand 
> wrote:
> 
>> Removing laurel's email address as it appears to be wrong.
>> 
>> -s
>> 
>> On Wed, Nov 2, 2016 at 10:49 AM, siddharth anand <
>> san...@apache.org>
>> wrote:
>> 
>>> +1 for that idea. We should place all links on the wiki and just
>>> have
> the
>>> project page point to the wiki!
>>> (https://airflow.incubator.apache.org/project.html).
>>> 
>>> Gerard, would you like to file a quick PR for that change and I
>> can
>>> approve/merge it?
>>> -s
>>> 
>>> On Wed, Nov 2, 2016 at 10:37 AM, Gerard Toonstra <
 gtoons...@gmail.com>
>>> wrote:
>>> 
 They are both on the project page of the airflow documentation
>> in
 resources
 & links and on the wiki, the wiki is a bit
 richer in that regard. Maybe link to the wiki from the doc pages
>> instead,
 so it's all in one place?
 
 https://cwiki.apache.org/confluence/display/AIRFLOW/
>> Airflow+Links
 
 https://airflow.incubator.apache.org/project.html
 
 G>
 
 
 
 On Wed, Nov 2, 2016 at 5:13 PM, Maxime Beauchemin <
 maximebeauche...@gmail.com> wrote:
 
> Hi,
> 
> Laurel Brunk reached out to share this great blog post he
>> wrote
> about
 using
> Airflow at Astronomer:
> http://www.astronomer.io/blog/airflow-at-astronomer
> 
> Where should we link out to this type of material? README.md?
>>> The
> Confluence wiki?
> 
> Thanks,
> 
> Max
> 
 
>>> 
>>> 
>> 
> 
 
>>> 
>> 

Re: Airflow documentation @ apache.org

[Hitesh Shah]

"© Copyright 2014, Maxime Beauchemin, Airbnb.” should be fixed as soon as 
possible. 

Please take a look at the ASF branding guidelines as well as other 
incubator/podling websites on the various trademark/disclaimer comments that 
should be mentioned on the site. 

thanks
— Hitesh 

> On Jun 6, 2016, at 11:58 AM, Maxime Beauchemin  
> wrote:
> 
> Airflowers,
> 
> A new version of the documentation that was available
> pythonhosted.org/airflow/ is now also up here:
> http://airflow.incubator.apache.org/
> 
> Information as to how to build and deploy the docs is here:
> https://cwiki.apache.org/confluence/display/AIRFLOW/Building+and+deploying+the+docs
> 
> Considering SEO and to minimize edits and confusion, I'd prefer keeping `
> pythonhosted.org` as the main entry point until we get our final `
> airflow.incubator.apache.org` subdomain.
> 
> Cheers!
> 
> Max

Re: Upcoming Podling report, due July 1st

[Hitesh Shah]

Additionally, it would be good if someone from the community can keep 
http://incubator.apache.org/projects/airflow.html updated for important events 
such as new committers, infra setup, code migration, etc. 


— Hitesh 

> On May 27, 2016, at 5:35 PM, siddharth anand  wrote:
> 
> Folks,
> I have updated the June Podling report for Airflow
> https://wiki.apache.org/incubator/June2016
> 
> Please review and share any thoughts before June 1.
> -s
> 
> On Wed, May 25, 2016 at 7:09 PM, siddharth anand  wrote:
> 
>> I can.
>> 
>> Sent from Sid's iPhone
>> 
>>> On May 25, 2016, at 7:18 PM, Jakob Homan  wrote:
>>> 
>>> Just a reminder that the second Podling report is imminent.  After
>>> three monthly reports (this is our second), we move to quarterly
>>> reporting.  Anyone want to volunteer?
>>> 
>>> http://wiki.apache.org/incubator/June2016
>>> 
>>> -Jakob
>> 

Re: Announcing the long awaited 1.7.1 Release

[Hitesh Shah]

Given that this release is not an ASF voted release, can we please make sure 
that there is no confusion regarding that i.e. there are very clear 
docs/announcements that 1.7.1 is *not* a release from ASF incubation. 

For example, "https://github.com/apache/incubator-airflow/releases/tag/1.7.1” 
should not exist. Additionally, I had assumed that 1.7.1 would be tagged with 
1.7.1-airbnb or something similar to reduce the confusion.
 
thanks
— Hitesh

> On May 20, 2016, at 1:51 PM, Chris Riccomini  wrote:
> 
> Special thanks to Dan, Arthur, and Maxime for doing a lot of the burn-in
> work and validation on their clusters! Very excited to try this out.
> 
> Also, when should we start on the next release? This one took so long that
> there's actually a fair amount of stuff already on master that's not in
> 1.7.1. Plus, if we expect 3-5 RCs, it might be better to start earlier
> rather than later.
> 
> On Fri, May 20, 2016 at 11:23 AM, siddharth anand  wrote:
> 
>> Hi Folks!
>> Release 1.7.1 is out with large bunch of stability fixes & new features -
>> 214 commits to be precise as per CHANGELOG.txt
>> 
>> 
>> You can install via Pypi  or
>> via Git tag <
>> https://github.com/apache/incubator-airflow/releases/tag/1.7.1>
>> 
>> Special thanks goes to Dan Davydov @aoen  for
>> shepherding this release, which has been a beast!
>> 
>> -s (Sid) on behalf of your Apache Airflow Maintainers
>> 
>> 

Re: Voting Changes for Scheduler-related PRs/Commits

[Hitesh Shah]

Instead of trying to formalize a set of committers who have superior rights 
compared to other committers ( which Jakob has been trying to point out is a 
big no-no ), why not just trust committers to do the right thing? If there is a 
contribution to some sensitive aspects of the code, only a committer who is 
well versed with those bits will review and commit the patch. If the project 
committee decides to vote someone in as a committer, they should trust that 
committer to do what is right for the project. It should not be the case that 
the committer is voted in but is not then not allowed commit code to certain 
parts of the project due to additional rights that he/she has not yet acquired. 
And yes, sure they will be mistakes but the incubation process does involve the 
growing the community and understanding how to manage and govern the project as 
the community grows ( without introducing any hierarchies ). 
 
thanks
— Hitesh


> On May 12, 2016, at 9:56 PM, Bolke de Bruin  wrote:
> 
> Hi,
> 
> I think we can approach this a bit differently. Apache is indeed a 
> meritocracy, ie. based on the ability and talents of the individuals within 
> the community. I don’t think this means every committer has the same set of 
> abilities and talents. For example, although I consider myself very talented 
> (…) I don’t think I have the ability to assess all consequences I make to the 
> scheduler. Here I would like to have Max, Dan, Paul and/or Jeremiah involved.
> 
> So here in lies the solution I think. For changes to the core components 
> scheduler and executor I suggest the following. Large changes: +1 required 
> from 2 persons from the following list: Dan, Max, Paul, Jeremiah. Minor 
> changes: +1 from Dan, Max, Paul, Jeremiah, Sid, Bolke. The one who proposes 
> the change is excluded from voting. In order to increase diversity these 
> lists are reviewed by the community at least every 6 months. 
> 
> We would catch two birds with one stone: It addresses the concern of 
> stability in the core and we have a path by which we can increase diversity 
> (and we set the first step by including Jeremiah).
> 
> Obviously, by having better tests and improved coverage we can lower the bar 
> (ie. the required ability) to make changes to the core components. We are not 
> there yet, so I would invite anyone aspiring to make changes to the core to 
> write a couple of tests first ;-).
> 
> What do you think?
> 
> Cheers
> Bolke
> 
> 
> 
> Firstly, we are indeed building an Apache community. This involves a company, 
> in this case Airbnb, 
> 
>> Op 13 mei 2016, om 00:14 heeft Jakob Homan  het volgende 
>> geschreven:
>> 
>> Dan-
>>  I'm afraid not, since that's just a way evading the Apache Way
>> rather than working towards it, as Incubation is meant to do.  (Here's
>> a good presentation for those unfamiliar with this manta:
>> http://www.slideshare.net/gagravarr/the-apache-way-apachecon-2014
>> You'll hear it a lot here).
>>  The concern here is that bad code may make it into the source tree,
>> and eventually into a release.  First, I'd say, yeah, that'll happen.
>> All code's reversible, so we can't guarantee it won't happen.  But
>> there are a few things the community can do:
>> * Maintain a devel branch that interested parties could run off of.
>> This would give time for features to be tested before being merged to
>> master.  Some policy of merging devel to master could be in place.
>> * Switch to a Commit-then-Review model (any committer can commit a
>> patch without a +1; this makes reverting bad commits easy and routine
>> without the drama/conflict often associated with reverts in
>> Review-then-Commit projects).
>> * Improve test coverage and utilize ASF and Github resources for testing.
>> 
>> What we can't do is tie abilities/privileges/responsibilities to
>> companies (or only people who work for certain companies).  The big
>> goal of Incubation is to develop a healthy community around the code
>> that can survive and thrive even if one group of contributors
>> disappear (say, if a company decides to pull people off the project).
>> This is why you'll often see big, big arguments around PMC/podling
>> diversity flare up (e.g. http://bit.ly/ASFWayDiversityArgument).
>> 
>> -Jakob
>> 
>> On 12 May 2016 at 14:47, Dan Davydov  wrote:
>>> @Jakob
>>> What if we made it more generic, e.g. a +1 from any commiter from a company
>>> that is running at a certain scale (e.g. at least X workers) and willing to
>>> help stage releases in their prods until we have more comprehensive test
>>> coverage/an open source staging environment? This is in Airflow's best
>>> interests as otherwise stability will suffer.
>>> 
>>> On Thu, May 12, 2016 at 1:44 PM, Chris Riccomini 
>>> wrote:
>>> 
 @Sid, perhaps defining a cool-off window before a scheduler change can be
 committed. That way, everyone that cares can have 

Re: Git code migration

[Hitesh Shah]

Yes - the code does need to be migrated now.

Migration on graduation is pretty straightforward too. There will be some very 
minor pains for in-flight pull requests but that should be easily managed as 
the code itself won’t be changing and just the repo. 

thanks
— Hitesh

On Apr 21, 2016, at 5:15 PM, Chris Riccomini  wrote:

> Hey Jakob/Hitesh,
> 
> One question that did come up about code migration was whether we need to
> migrate the code now, or whether we can wait until graduation. I'm nearly
> 100% certain that we need to migrate now.
> 
> One gripe with this is that the repo is currently `incubator-airflow`, but
> will presumably just become `airflow` after graduation, which means ANOTHER
> migration. I recall the graduation migration being fairly light-weight
> (they just rename the repo), but I wanted to follow up on this to check.
> 
> Cheers,
> Chris