Re: kwargs usage in BaseOperator

[Craig Rodrigues]

On Monday, May 21, 2018, Tao Feng  wrote:

>
> I wonder whether we still have the plan to remove kwargs in airflow 2.0(do
> we have the timeline for airflow 2.0?) as the warning generates too many
> noises for our internal production env as well as the travis CI log.
>
>
>

This may be annoying for your logs, but you should spend the time to fix
your code to not generate these warnings.  I recently upgraded my system
from airflow 1.7.0 to airflow 1.9.0 and had to fix a lot of DAG code to not
print DeprecationWarnings.  It was a lot of work, but it was worth it.

--
Craig

Re: kwargs usage in BaseOperator

[Tao Feng]

Ping. Anyone has any idea on the question(whether airflow will remove
kwargs in BaseOperator)?

On Mon, May 21, 2018 at 11:26 AM, Tao Feng  wrote:

> Hi,
>
> I have a question regarding kwargs usage in BaseOperator(https://github.
> com/apache/incubator-airflow/blob/master/airflow/models.py#L2306-L2315).
> Since this pr(https://github.com/apache/incubator-airflow/pull/1285)
> checked in, airflow turns on deprecation warning by default. And per the
> comment, the community plans to remove kwargs usage in airflow 2.0.
>
> I wonder whether we still have the plan to remove kwargs in airflow 2.0(do
> we have the timeline for airflow 2.0?) as the warning generates too many
> noises for our internal production env as well as the travis CI log.
>
> Thanks,
> -Tao
>

Re: Moving to Github? Re: Merging PRs, closing Jira tickets (a.k.a New Committer) guide?

[Driesprong, Fokko]

Are we moving to Gitbox? I would really like that. I see a lot of other
project moving to Gitbox at the moment (Flink, Parquet). How's your
schedule Ash? ;)

Cheers, Fokko

2018-03-09 18:53 GMT+01:00 Bolke de Bruin :

> I would love to be able to close PRs on GitHub, but personally I’m quite
> ok with JIRA. GitHub issues tend to get messy imho. I also like that it is
> clear from the subject of Pr what the associated issues was, since we moved
> history became a lot cleaner and changelogs are now easy to generate.
>
> My 2cents
>
> B.
>
> Verstuurd vanaf mijn iPad
>
> > Op 9 mrt. 2018 om 18:01 heeft Ash Berlin-Taylor 
> het volgende geschreven:
> >
> > That sounds like it is at worth me at least coming up a proposal for us
> to vote on then..
> >
> > One thing that might help with the "target version" is multiple Github
> Projects[1]: -- that, or labels, are the only way for a github issue to be
> in "two" groups at the same time.
> >
> > I'll see what I can do, but make zero promises due to imminent
> baby-driven "sleep" schedule ;)
> >
> > [1]: https://help.github.com/articles/about-project-boards/
> >
> >
> >> On 9 Mar 2018, at 16:10, Maxime Beauchemin 
> wrote:
> >>
> >> We use Gitbox and no Jira for Apache Superset and are happy with it.
> >>
> >> One downside is around the current release management tooling for
> Airflow
> >> has bindings with Jira and the "target version" field.
> >>
> >> Max
> >>
> >> On Thu, Mar 8, 2018 at 6:31 AM, Ash Berlin-Taylor <
> >> ash_airflowl...@firemirror.com> wrote:
> >>
> >>> I've done a bit of digging and there's an Apache "project" called
> >>> gitbox[1] that, if we choose to go that way lets us use Github more
> >>> "natively".
> >>>
> >>> The BookKeeper project migrated to using Github exclusively lsat Jun[2]
> >>> and from the looks of their Github repo are still using this approach,
> and
> >>> their Jira is read only. Their proposal on the migration was
> >>> https://cwiki.apache.org/confluence/display/BOOKKEEPER/
> >>> BP-9+-+Github+issues+for+Issue+Tracking
> >>>
> >>> I think there are three ways we could go:
> >>>
> >>> 1. Nothing changes, we stay as we are and commit to the ASF git repo.
> >>> 2. Move to Gitbox and commit directly to githb, keep issues in Jira.
> >>> 3. Do as BookKeeper did and move to using Github Issues as well as
> Gitub
> >>> for the repo.
> >>>
> >>> Is there interest from anyone else in 2 or 3, if so I will attempt to
> draw
> >>> up a more detailed proposal.
> >>>
> >>> [1]: https://lists.apache.org/thread.html/Znkiyqnxqzryecv
> >>> [2]: http://mail-archives.apache.org/mod_mbox/bookkeeper-dev/
> 201706.mbox/%
> >>> 3CCAO2yDybRq2VUM1JYo_6VT_H8Ca7Lu8af6H-2CZKQzYT6xYGU-g%40mail.gmail.com
> %3E
> >>>
> >>>
>  On 6 Mar 2018, at 09:57, Ash Berlin-Taylor
>  >>> com> wrote:
> 
>  Ah that would explain why I don't have a button :)
> 
>  Is this Apache policy, or is it possible for committers to be granted
> >>> permission to do this? Having this permission would also let us click
> the
> >>> "rerun tests" button in Travis which would be nice.
> 
>  Is it worth opening an INFRA ticket asking for this, or is it not
> >>> possible?
> 
>  -ash
> 
> > On 6 Mar 2018, at 08:25, Driesprong, Fokko 
> >>> wrote:
> >
> > Hi Ash,
> >
> > As a committer we don't have any rights on the Github itself. The
> Github
> > repo is just a sync of the apache repo. Unfortunately, therefore we
> >>> don't
> > have the right to close any PR.
> >
> > Cheers, Fokko
> >
> > 2018-03-06 0:49 GMT+01:00 Ash Berlin-Taylor <
> >>> ash_airflowl...@firemirror.com>
> > :
> >
> >> I've merged two PRs now, and the second one seemed to be better
> >>> (though I
> >> did have some trouble with the tool not merging properly and I
> needed
> >>> to
> >> manually coax git. Hmm)
> >>
> >> Jira: I _think_ that previously I could only comment on Jira issues.
> >>> With
> >> the new permissions I can now do more (as those of you subscribed to
> >>> the
> >> commit list will see) -- I started going through old Jira tickets
> and
> >> closing ones that are no longer an issue or that were fixed+merged
> but
> >>> not
> >> closed.
> >>
> >> Github: I don't have a button to close PRs in Github - Is that
> >>> expected?
> >>
> >> -ash
> >>
> >>
> >>> On 4 Mar 2018, at 00:59, Sid Anand  wrote:
> >>>
> >>> Hi Ash,
> >>> Welcome aboard.
> >>>
> >>> Firstly, I'm moving this conversation over to the dev list -- the
> >>> first
> >>> lesson we all learned at the insistence of the incubator mentors
> was
> >>> to
> >> use
> >>> the private list for voting and discussion on PMC matters. They
> >>> require
> >>> that all information-oriented discussions be routed to the
> 

Re: Chicago Airflow user meetup group

[Sid Anand]

The 4th one is not required. You are free to add anyone you wish as admins.
As a best practice, ensure there are enough admins to avoid a dead/unowned
meetup.

-s

On Tue, May 22, 2018 at 9:13 AM, Victor Noagbodji <
vnoagbo...@amplify-nation.com> wrote:

> I am curious about this one. Why should committers be added as admins?
>
> > On May 22, 2018, at 12:08 PM, Sid Anand  wrote:
> >
> > That's great news and we welcome the new Meetup. Here are a few pointers:
> >
> >   1. Add your new meet-up on :
> >   https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups
> >   2. Add the new Meetup to
> >   https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements &
> >   follow the guidelines on the link above to invite speakers and
> attendees to
> >   your meet-up.
> >  1. Once you have done 1 & 2, we can tweet your new Meetup on the
> >  Apache Airflow twitter account
> >   3. To do 1&2, you need to give me your cwiki id so that I can give your
> >   edit perms
> >   4. Finally, add some of the committers to the meetup as admins
> >  1. You can add me (https://www.meetup.com/members/10527138/) and
> any
> >  other committers that voice interest!
> >
> >
> >
> > -s
> >
> > On Mon, May 21, 2018 at 9:26 PM, Trevor Beuthel 
> wrote:
> >
> >> Hi there,
> >>
> >> My name's Trevor Beuthel and I would like to start an airflow meetup
> group
> >> in Chicago.
> >>
> >> I work at a company called Vibes here in Chicago, and we just rolled
> out a
> >> decent sized migration to airflow that took us a little under a year to
> >> complete. We use many of the built in features of airflow and also
> created
> >> many plugins to satisfy our needs.
> >>
> >> I've been speaking at a few events with Snowflake, and there seems to
> be a
> >> lot of interest in airflow out here. A handful of companies that I've
> >> talked to are actually using it, many are experimenting with it, and
> much
> >> more have heard of it and are interested.
> >>
> >> Let me know if I have your blessing to create the group and the steps I
> >> should take to do so.
> >>
> >> Thanks,
> >> Trevor
> >>
>
>

Re: Where to put docs on configuring specific kinds of connections? (Or restructuring the docs the way Django does)

[Tim Swast]

PR: https://github.com/apache/incubator-airflow/pull/3400
Jira: https://issues.apache.org/jira/browse/AIRFLOW-2509


On Tue, May 22, 2018 at 9:12 AM Tim Swast  wrote:

> I think the whole "Configuration" page could benefit from getting split
> into separate how-to guides. Basically, every sub-heading in
> Configuration becomes a different how-to guide. Since the configuration
> page is attempting to do double-duty as a tutorial right now, keep the
> current sequence, at least until a "Deploying a Production Airflow
> Environment" tutorial is written.
>
> I started this split (PR coming soon). I think the split document makes it
> clearer where some additional guidance would be useful:
>
>- Connections: could use some specific examples for how to created
>different kinds of connections.
>- Database backends: could use some specific examples on configuring a
>MySQL or Postgres database backend.
>
> *  •  **Tim Swast*
> *  •  *Software Friendliness Engineer
> *  •  *Google Cloud Developer Relations
> *  •  *Seattle, WA, USA
>
>
> On Mon, May 21, 2018 at 10:20 PM Taylor Edmiston 
> wrote:
>
>> Hey Tim -
>>
>> I came to Airflow from the Django world as well and had the same thought
>> that much of the work that's been put into their docs over time could be
>> applied here too.  In terms of documentation for large Python projects,
>> perhaps they're the gold standard.
>>
>> Can you give a few examples of the specific docs you think would benefit
>> from refactoring out here?  I'd be happy to assist with this effort as well.
>>
>> Taylor
>> Sent from my iPhone
>>
>> > On May 21, 2018, at 4:35 PM, Tim Swast 
>> wrote:
>> >
>> > Hey folks,
>> >
>> > I'd like to write some docs on how to create a GCP connection (and leave
>> > room for documenting other kinds of connections as well). Currently it
>> > seems like there are a couple places such a thing could fit:
>> >
>> >   - https://airflow.incubator.apache.org/configuration.html#connections
>> >   -
>> >
>> https://airflow.incubator.apache.org/integration.html#gcp-google-cloud-platform
>> >
>> > There's also the concepts guide, but I definitely don't think that's the
>> > right place for documenting a specific task like this.
>> >
>> > There's a principle I'm used to following with the GCP docs, that the
>> distinct
>> > kinds of documentation
>> > <
>> http://www.writethedocs.org/videos/eu/2017/the-four-kinds-of-documentation-and-why-you-need-to-understand-what-they-are-daniele-procida/
>> >
>> > should be organized separately. The Django project does this
>> > https://docs.djangoproject.com/en/2.0/ by splitting into
>> >
>> >   - Tutorials
>> >   - Topic guides (what Airflow calls Concepts)
>> >   - Reference guides
>> >   - How-to guides
>> >
>> > I'd like to propose we split some of the existing "Configuration" topics
>> > into separate how-to guides. What do you think?
>> >
>> > Meta: Should I create JIRA issues for this kind of pre-discussion or
>> start
>> > here as I've done?
>> >
>> > *  •  **Tim Swast*
>> > *  •  *Software Friendliness Engineer
>> > *  •  *Google Cloud Developer Relations
>> > *  •  *Seattle, WA, USA
>>
>

Re: Chicago Airflow user meetup group

[Victor Noagbodji]

I am curious about this one. Why should committers be added as admins?

> On May 22, 2018, at 12:08 PM, Sid Anand  wrote:
> 
> That's great news and we welcome the new Meetup. Here are a few pointers:
> 
>   1. Add your new meet-up on :
>   https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups
>   2. Add the new Meetup to
>   https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements &
>   follow the guidelines on the link above to invite speakers and attendees to
>   your meet-up.
>  1. Once you have done 1 & 2, we can tweet your new Meetup on the
>  Apache Airflow twitter account
>   3. To do 1&2, you need to give me your cwiki id so that I can give your
>   edit perms
>   4. Finally, add some of the committers to the meetup as admins
>  1. You can add me (https://www.meetup.com/members/10527138/) and any
>  other committers that voice interest!
> 
> 
> 
> -s
> 
> On Mon, May 21, 2018 at 9:26 PM, Trevor Beuthel  wrote:
> 
>> Hi there,
>> 
>> My name's Trevor Beuthel and I would like to start an airflow meetup group
>> in Chicago.
>> 
>> I work at a company called Vibes here in Chicago, and we just rolled out a
>> decent sized migration to airflow that took us a little under a year to
>> complete. We use many of the built in features of airflow and also created
>> many plugins to satisfy our needs.
>> 
>> I've been speaking at a few events with Snowflake, and there seems to be a
>> lot of interest in airflow out here. A handful of companies that I've
>> talked to are actually using it, many are experimenting with it, and much
>> more have heard of it and are interested.
>> 
>> Let me know if I have your blessing to create the group and the steps I
>> should take to do so.
>> 
>> Thanks,
>> Trevor
>> 

Re: Where to put docs on configuring specific kinds of connections? (Or restructuring the docs the way Django does)

[Tim Swast]

I think the whole "Configuration" page could benefit from getting split
into separate how-to guides. Basically, every sub-heading in Configuration
becomes a different how-to guide. Since the configuration page is
attempting to do double-duty as a tutorial right now, keep the current
sequence, at least until a "Deploying a Production Airflow Environment"
tutorial is written.

I started this split (PR coming soon). I think the split document makes it
clearer where some additional guidance would be useful:

   - Connections: could use some specific examples for how to created
   different kinds of connections.
   - Database backends: could use some specific examples on configuring a
   MySQL or Postgres database backend.

*  •  **Tim Swast*
*  •  *Software Friendliness Engineer
*  •  *Google Cloud Developer Relations
*  •  *Seattle, WA, USA


On Mon, May 21, 2018 at 10:20 PM Taylor Edmiston 
wrote:

> Hey Tim -
>
> I came to Airflow from the Django world as well and had the same thought
> that much of the work that's been put into their docs over time could be
> applied here too.  In terms of documentation for large Python projects,
> perhaps they're the gold standard.
>
> Can you give a few examples of the specific docs you think would benefit
> from refactoring out here?  I'd be happy to assist with this effort as well.
>
> Taylor
> Sent from my iPhone
>
> > On May 21, 2018, at 4:35 PM, Tim Swast  wrote:
> >
> > Hey folks,
> >
> > I'd like to write some docs on how to create a GCP connection (and leave
> > room for documenting other kinds of connections as well). Currently it
> > seems like there are a couple places such a thing could fit:
> >
> >   - https://airflow.incubator.apache.org/configuration.html#connections
> >   -
> >
> https://airflow.incubator.apache.org/integration.html#gcp-google-cloud-platform
> >
> > There's also the concepts guide, but I definitely don't think that's the
> > right place for documenting a specific task like this.
> >
> > There's a principle I'm used to following with the GCP docs, that the
> distinct
> > kinds of documentation
> > <
> http://www.writethedocs.org/videos/eu/2017/the-four-kinds-of-documentation-and-why-you-need-to-understand-what-they-are-daniele-procida/
> >
> > should be organized separately. The Django project does this
> > https://docs.djangoproject.com/en/2.0/ by splitting into
> >
> >   - Tutorials
> >   - Topic guides (what Airflow calls Concepts)
> >   - Reference guides
> >   - How-to guides
> >
> > I'd like to propose we split some of the existing "Configuration" topics
> > into separate how-to guides. What do you think?
> >
> > Meta: Should I create JIRA issues for this kind of pre-discussion or
> start
> > here as I've done?
> >
> > *  •  **Tim Swast*
> > *  •  *Software Friendliness Engineer
> > *  •  *Google Cloud Developer Relations
> > *  •  *Seattle, WA, USA
>

Re: Chicago Airflow user meetup group

[Victor Noagbodji]

Hey Trevor, meetup.com is the only service I used and know 
helps with meetups. I don't live in Chicago, so I don't know what other 
restrictions there are. Wish you the best! Oh, +1 for putting streams on 
YouTube : )

On May 22, 2018, at 12:26 AM, Trevor Beuthel 
> wrote:

Hi there,

My name's Trevor Beuthel and I would like to start an airflow meetup group
in Chicago.

I work at a company called Vibes here in Chicago, and we just rolled out a
decent sized migration to airflow that took us a little under a year to
complete. We use many of the built in features of airflow and also created
many plugins to satisfy our needs.

I've been speaking at a few events with Snowflake, and there seems to be a
lot of interest in airflow out here. A handful of companies that I've
talked to are actually using it, many are experimenting with it, and much
more have heard of it and are interested.

Let me know if I have your blessing to create the group and the steps I
should take to do so.

Thanks,
Trevor

Re: Chicago Airflow user meetup group

[Sid Anand]

That's great news and we welcome the new Meetup. Here are a few pointers:

   1. Add your new meet-up on :
   https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups
   2. Add the new Meetup to
   https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements &
   follow the guidelines on the link above to invite speakers and attendees to
   your meet-up.
  1. Once you have done 1 & 2, we can tweet your new Meetup on the
  Apache Airflow twitter account
   3. To do 1&2, you need to give me your cwiki id so that I can give your
   edit perms
   4. Finally, add some of the committers to the meetup as admins
  1. You can add me (https://www.meetup.com/members/10527138/) and any
  other committers that voice interest!



-s

On Mon, May 21, 2018 at 9:26 PM, Trevor Beuthel  wrote:

> Hi there,
>
> My name's Trevor Beuthel and I would like to start an airflow meetup group
> in Chicago.
>
> I work at a company called Vibes here in Chicago, and we just rolled out a
> decent sized migration to airflow that took us a little under a year to
> complete. We use many of the built in features of airflow and also created
> many plugins to satisfy our needs.
>
> I've been speaking at a few events with Snowflake, and there seems to be a
> lot of interest in airflow out here. A handful of companies that I've
> talked to are actually using it, many are experimenting with it, and much
> more have heard of it and are interested.
>
> Let me know if I have your blessing to create the group and the steps I
> should take to do so.
>
> Thanks,
> Trevor
>

Re: Chicago Airflow user meetup group

[Vasyl Harasymiv]

Hi Trevor,

Great idea! I saw you speaking at Snowflake event. We are using Airflow and
are ready to offer space
at ActiveCampaign in Chicago and arrange for live streaming (provided it is
not booked out for a specific
evening already) or join at your location (we are at 1 North Dearborn).

Mark/Conrad/Vasyl/Scott/Ben @AC



On Tue, May 22, 2018 at 10:59 AM Trevor Beuthel  wrote:

> Hi there,
>
> My name's Trevor Beuthel and I would like to start an airflow meetup group
> in Chicago.
>
> I work at a company called Vibes here in Chicago, and we just rolled out a
> decent sized migration to airflow that took us a little under a year to
> complete. We use many of the built in features of airflow and also created
> many plugins to satisfy our needs.
>
> I've been speaking at a few events with Snowflake, and there seems to be a
> lot of interest in airflow out here. A handful of companies that I've
> talked to are actually using it, many are experimenting with it, and much
> more have heard of it and are interested.
>
> Let me know if I have your blessing to create the group and the steps I
> should take to do so.
>
> Thanks,
> Trevor
>

Chicago Airflow user meetup group

[Trevor Beuthel]

Hi there,

My name's Trevor Beuthel and I would like to start an airflow meetup group
in Chicago.

I work at a company called Vibes here in Chicago, and we just rolled out a
decent sized migration to airflow that took us a little under a year to
complete. We use many of the built in features of airflow and also created
many plugins to satisfy our needs.

I've been speaking at a few events with Snowflake, and there seems to be a
lot of interest in airflow out here. A handful of companies that I've
talked to are actually using it, many are experimenting with it, and much
more have heard of it and are interested.

Let me know if I have your blessing to create the group and the steps I
should take to do so.

Thanks,
Trevor

Re: celery problem: cannot override celery_broker_transport_options

[Ash Berlin-Taylor]

To use with the SQLA backend to celery you need to override the options Airflow 
passes to Celery. Those come from 
https://github.com/apache/incubator-airflow/blob/v1-10-test/airflow/config_templates/default_celery.py

Since you don't want most/all of those options (and there is no way in the 
config file to _remove_ a setting) you will have to point airflow to a 
different file for the celery config:

This line in the config is what you will need to change:

# Import path for celery configuration options
celery_config_options = 
airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG

If you create something like config/celery_config.py containing:


CELERY_CONFIG = {
# Just the options you want to set
}


(config/ should exist along side your dags/ folder, and I think it should be 
added to the python path already). You can then set this in the config:

celery_config_options = celery_config.CELERY_CONFIG

That should give you complete control

> On 21 May 2018, at 09:50, Craig Rodrigues  wrote:
> 
> Hi,
> 
> I used this requirements.txt file to install airflow from the v1-10-test 
> branch:
> 
> git+https://github.com/celery/celery@master#egg=celery
> git+https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
> kombu>=4.1.0
> 
> 
> In my airflow.cfg, I have:
> 
> [celery]
> executor = CeleryExecutor
> 
> executor = CeleryExec
> broker_url = sqla+mysql://airflow:blah@localhost:3306/mydb
> 
> [celery_broker_transport_options]
> #
> #
> 
> However, if I manually run this code inside the webserver, I see:
> 
> python -c "from airflow import configuration; c = 
> configuration.conf.getsection('celery_broker_transport_options'); print(c)"
> OrderedDict([(u'visibility_timeout', 21600), (u'ssl_active', False), 
> (u'ssl_key', u''), (u'ssl_cert', u''), (u'ssl_cacert', u'')])
> 
> My worker crashes with this error:
> 
> 
> [2018-05-21 07:46:12,406] {configuration.py:212} WARNING - section/key 
> [celery/ssl_active] not found in config
> [2018-05-21 07:46:12,407] {default_celery.py:51} WARNING - Celery Executor 
> will run without SSL
> [2018-05-21 07:46:12,411] {__init__.py:48} INFO - Using executor 
> CeleryExecutor
> [2018-05-21 07:46:13,086: CRITICAL/MainProcess] Unrecoverable error: 
> TypeError(u"Invalid argument(s) 
> 'ssl_key','ssl_cert','ssl_active','visibility_timeout','ssl_cacert' sent to 
> create_engine(), using configuration MySQLDialect_mysqldb/QueuePool/Engine.  
> Please check that the keyword arguments are appropriate for this combination 
> of components.",)
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/site-packages/celery/worker/worker.py", line 205, 
> in start
>self.blueprint.start(self)
>  File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in 
> start
>step.start(parent)
>  File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 369, in 
> start
>return self.obj.start()
>  File "/usr/lib/python2.7/site-packages/celery/worker/consumer/consumer.py", 
> line 322, in start
>blueprint.start(self)
>  File "/usr/lib/python2.7/site-packages/celery/bootsteps.py", line 119, in 
> start
>step.start(parent)
>  File "/usr/lib/python2.7/site-packages/celery/worker/consumer/tasks.py", 
> line 41, in start
>c.connection, on_decode_error=c.on_decode_error,
>  File "/usr/lib/python2.7/site-packages/celery/app/amqp.py", line 297, in 
> TaskConsumer
>**kw
>  File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 386, in 
> __init__
>self.revive(self.channel)
>  File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 408, in 
> revive
>self.declare()
>  File "/usr/lib/python2.7/site-packages/kombu/messaging.py", line 421, in 
> declare
>queue.declare()
>  File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 605, in declare
>self._create_queue(nowait=nowait, channel=channel)
>  File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 614, in 
> _create_queue
>self.queue_declare(nowait=nowait, passive=False, channel=channel)
>  File "/usr/lib/python2.7/site-packages/kombu/entity.py", line 649, in 
> queue_declare
>nowait=nowait,
>  File "/usr/lib/python2.7/site-packages/kombu/transport/virtual/base.py", 
> line 531, in queue_declare
>self._new_queue(queue, **kwargs)
>  File 
> "/usr/lib/python2.7/site-packages/kombu/transport/sqlalchemy/__init__.py", 
> line 82, in _new_queue
>self._get_or_create(queue)
>  File 
> "/usr/lib/python2.7/site-packages/kombu/transport/sqlalchemy/__init__.py", 
> line 70, in _get_or_create
>obj = self.session.query(self.queue_cls) \
>  File 
> "/usr/lib/python2.7/site-packages/kombu/transport/sqlalchemy/__init__.py", 
> line 65, in session
>_, Session = self._open()
>  File 
> "/usr/lib/python2.7/site-packages/kombu/transport/sqlalchemy/__init__.py", 
> line 56, in _open
>engine = 

Re: Where to put docs on configuring specific kinds of connections? (Or restructuring the docs the way Django does)

[Naik Kaxil]

Hi Tim,

My personal opinion is to write the docs on GCP connection down at GCP 
Integration page 

 would be an ideal fit but would like to hear others opinion as well.

+1 for creating separate How-To guides.

Regards,
Kaxil

On 21/05/2018, 21:35, "Tim Swast"  wrote:

Hey folks,

I'd like to write some docs on how to create a GCP connection (and leave
room for documenting other kinds of connections as well). Currently it
seems like there are a couple places such a thing could fit:

   - https://airflow.incubator.apache.org/configuration.html#connections
   -
   
https://airflow.incubator.apache.org/integration.html#gcp-google-cloud-platform

There's also the concepts guide, but I definitely don't think that's the
right place for documenting a specific task like this.

There's a principle I'm used to following with the GCP docs, that the 
distinct
kinds of documentation


should be organized separately. The Django project does this
https://docs.djangoproject.com/en/2.0/ by splitting into

   - Tutorials
   - Topic guides (what Airflow calls Concepts)
   - Reference guides
   - How-to guides

I'd like to propose we split some of the existing "Configuration" topics
into separate how-to guides. What do you think?

Meta: Should I create JIRA issues for this kind of pre-discussion or start
here as I've done?

*  •  **Tim Swast*
*  •  *Software Friendliness Engineer
*  •  *Google Cloud Developer Relations
*  •  *Seattle, WA, USA






Kaxil Naik 

Data Reply
2nd Floor, Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK 
phone: +44 (0)20 7730 6000
k.n...@reply.com
www.reply.com