Re: Postgres as a backend - no documentation on how to

[Shashank Chaturvedi]

Hello All,

I've created sample doc for my reference. You guys can check that doc and
share your feedback.

Thanks,
Shashank

On Tue, Feb 11, 2020 at 4:01 PM Ash Berlin-Taylor  wrote:

> Hi there,
>
> To answer the easiest one first -- pyscopg2. pip install
> apache-airflow[postgres]  will install psycopg2 for you, so.
>
> SQLAlchemy is the DB access layer Airflow uses, but I agree that there is
> an awful lot of assumed knowledge/context in the docs!
>
> Yes, before running airflow initdb  you will need to configure the
> permissions on the database. For postgres this is:
>
>
>- create the user/role in the DB. (either CREATE USER  sql statement,
>or createuser cli tool.)
>- create an empty DB for airflow. createdb --owner airflow airflow
>probably.
>- Then configure sql_alchemy_conn to point to this. For instance if
>the DB is on the local host then postgresql://airflow@/airflow
>
>
> On Feb 11 2020, at 9:35 am, /dev /local/ca  wrote:
>
> After going over the documentation for installing Airflow, and using
> Postgres as a backend, I am really confused.
>
> There is no guide or step by step installation manual, just some random
> things, ideas and fragments of memory that someone splat down on a web page.
>
> ---
> I am led to believe that I need to configure things with postgres before
> running `airflow initdb`, but that is not clear in the documentation.
>
> Do I create a role/user first before using Airflow? and specify that
> somewhere in a config file before running 'airflow initdb'
>
> --
> Also the instructions jump right in talking about something called
> 'sql_alchemy' as if I should know what that is, but I don't.
>
> In addition it says: If you decide to use *Postgres*, we recommend using
> the psycopg2 driver , and again, no instructions there as to what that
> means.
>
> ---
> I am more than willing to help the project write clear, concise
> documentation and communicate with others, as it seems no one on the team
> has developed this skill set, that can be put to use to help others get
> this installed.
>
> If someone could Jesus please take just five minutes and document this so
> I do not waste 3-4 hours doing research, subscribing to user lists and
> being misled by the drive by geniuses that respond on Stackoverflow it
> would be greatly appreciated.
>
>

-- 
Thanks,
Shashank


Airflow with Celery Executor,Postgres and Rabbitmq.docx
Description: MS-Word 2007 document

Re: Postgres as a backend - no documentation on how to

[Ash Berlin-Taylor]

Hi there,

To answer the easiest one first -- pyscopg2. pip install 
apache-airflow[postgres] will install psycopg2 for you, so.
SQLAlchemy is the DB access layer Airflow uses, but I agree that there is an 
awful lot of assumed knowledge/context in the docs!
Yes, before running airflow initdb you will need to configure the permissions 
on the database. For postgres this is:
create the user/role in the DB. (either CREATE USER sql statement, or 
createuser cli tool.)
create an empty DB for airflow. createdb --owner airflow airflow probably.

Then configure sql_alchemy_conn to point to this. For instance if the DB is on 
the local host then postgresql://airflow@/airflow

On Feb 11 2020, at 9:35 am, /dev /local/ca  wrote:
> After going over the documentation for installing Airflow, and using Postgres 
> as a backend, I am really confused.
>
> There is no guide or step by step installation manual, just some random 
> things, ideas and fragments of memory that someone splat down on a web page.
>
> ---
> I am led to believe that I need to configure things with postgres before 
> running `airflow initdb`, but that is not clear in the documentation.
>
> Do I create a role/user first before using Airflow? and specify that 
> somewhere in a config file before running 'airflow initdb'
>
> --
> Also the instructions jump right in talking about something called 
> 'sql_alchemy' as if I should know what that is, but I don't.
>
> In addition it says: If you decide to use Postgres, we recommend using the 
> psycopg2 driver , and again, no instructions there as to what that means.
>
> ---
> I am more than willing to help the project write clear, concise documentation 
> and communicate with others, as it seems no one on the team has developed 
> this skill set, that can be put to use to help others get this installed.
>
> If someone could Jesus please take just five minutes and document this so I 
> do not waste 3-4 hours doing research, subscribing to user lists and being 
> misled by the drive by geniuses that respond on Stackoverflow it would be 
> greatly appreciated.

Re: Postgres as a backend - no documentation on how to

[Jarek Potiuk]

Hello,

Thank you for your comment. We share your pain, improving the docs is an
ongoing process and we are more than happy if you can help. Airflow is a
community driven project, so everyone is welcome to contribute to the
documentation.

In fact in our "documentation" (including installation page
https://airflow.apache.org/docs/stable/installation.html). There is a link
at the bottom that allows you to go to github, fork it and see where the
page is and suggest improvements.

We are more than happy if you can help and describe it much better than it
is currently.

J.

On Tue, Feb 11, 2020 at 10:35 AM /dev /local/ca 
wrote:

> After going over the documentation for installing Airflow, and using
> Postgres as a backend, I am really confused.
>
> There is no guide or step by step installation manual, just some random
> things, ideas and fragments of memory that someone splat down on a web page.
>
> ---
> I am led to believe that I need to configure things with postgres before
> running `airflow initdb`, but that is not clear in the documentation.
>
> Do I create a role/user first before using Airflow? and specify that
> somewhere in a config file before running 'airflow initdb'
>
> --
> Also the instructions jump right in talking about something called
> 'sql_alchemy' as if I should know what that is, but I don't.
>
> In addition it says: If you decide to use *Postgres*, we recommend using
> the psycopg2 driver , and again, no instructions there as to what that
> means.
>
> ---
> I am more than willing to help the project write clear, concise
> documentation and communicate with others, as it seems no one on the team
> has developed this skill set, that can be put to use to help others get
> this installed.
>
> If someone could Jesus please take just five minutes and document this so
> I do not waste 3-4 hours doing research, subscribing to user lists and
> being misled by the drive by geniuses that respond on Stackoverflow it
> would be greatly appreciated.
>


-- 

Jarek Potiuk
Polidea  | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea]