[
https://issues.apache.org/jira/browse/AIRFLOW-3385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
RIo updated AIRFLOW-3385:
-------------------------
Description:
I have set and up Airflow in Kubernetes. This Airflow using postgresql database
in other pods in same cluster.
When I try to run some dags that connect to same postgresql with configured
option, the connection to postgresql pods ask for OAUTH authorize
{code:java}
Reading local file:
/root/airflow/logs/anu_funnel_analysis_v2/task_4_get_bigquery_pandas/2018-06-29T02:30:00+00:00/1.log
[2018-11-23 15:56:22,796] {models.py:1335} INFO - Dependencies all met for
<TaskInstance: anu_funnel_analysis_v2.task_4_get_bigquery_pandas
2018-06-29T02:30:00+00:00 [queued]>
[2018-11-23 15:56:22,820] {models.py:1335} INFO - Dependencies all met for
<TaskInstance: anu_funnel_analysis_v2.task_4_get_bigquery_pandas
2018-06-29T02:30:00+00:00 [queued]>
[2018-11-23 15:56:22,821] {models.py:1547} INFO -
--------------------------------------------------------------------------------
Starting attempt 1 of 6
--------------------------------------------------------------------------------
[2018-11-23 15:56:22,838] {models.py:1569} INFO - Executing
<Task(PythonOperator): task_4_get_bigquery_pandas> on 2018-06-29T02:30:00+00:00
[2018-11-23 15:56:22,839] {base_task_runner.py:124} INFO - Running: ['bash',
'-c', u'airflow run anu_funnel_analysis_v2 task_4_get_bigquery_pandas
2018-06-29T02:30:00+00:00 --job_id 82 --raw -sd
DAGS_FOLDER/bigquery_for_funnel.py --cfg_path /tmp/tmpJ1Pe0O']
[2018-11-23 15:56:23,656] {base_task_runner.py:107} INFO - Job 82: Subtask
task_4_get_bigquery_pandas [2018-11-23 15:56:23,655] {settings.py:174} INFO -
setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
[2018-11-23 15:56:23,817] {base_task_runner.py:107} INFO - Job 82: Subtask
task_4_get_bigquery_pandas [2018-11-23 15:56:23,816] {__init__.py:51} INFO -
Using executor SequentialExecutor
[2018-11-23 15:56:23,952] {base_task_runner.py:107} INFO - Job 82: Subtask
task_4_get_bigquery_pandas [2018-11-23 15:56:23,950] {models.py:258} INFO -
Filling up the DagBag from /root/airflow/dags/bigquery_for_funnel.py
[2018-11-23 15:56:24,737] {base_task_runner.py:107} INFO - Job 82: Subtask
task_4_get_bigquery_pandas
/usr/local/lib/python2.7/dist-packages/airflow/contrib/operators/bigquery_operator.py:148:
DeprecationWarning: Deprecated parameter `bql` used in Task id:
task_2_bq_read_new_table_funnel. Use `sql` parameter instead to pass the sql to
be executed. `bql` parameter is deprecated and will be removed in a future
version of Airflow.
[2018-11-23 15:56:24,738] {base_task_runner.py:107} INFO - Job 82: Subtask
task_4_get_bigquery_pandas category=DeprecationWarning)
[2018-11-23 15:56:24,778] {base_task_runner.py:107} INFO - Job 82: Subtask
task_4_get_bigquery_pandas [2018-11-23 15:56:24,777] {cli.py:492} INFO -
Running <TaskInstance: anu_funnel_analysis_v2.task_4_get_bigquery_pandas
2018-06-29T02:30:00+00:00 [running]> on host airflow-5d87576f5b-v5qmg
[2018-11-23 15:56:24,837] {logging_mixin.py:95} INFO - [2018-11-23
15:56:24,837] {base_hook.py:83} INFO - Using connection to:
anudata-postgresql-service
[2018-11-23 15:56:25,335] {logging_mixin.py:95} INFO - Please visit this URL to
authorize this application:
https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=495642085510-k0tmvj2m941jhre2nbqka17vqpjfddtd.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fbigquery&state=zI3Uv6bt3ncdAGIPgWQ5PF36V3uMq0&prompt=consent&access_type=offline
{code}
Why this happen? in the same pods, one ask for OAUTH and other not?
was:
I have set and up Airflow in Kubernetes. This Airflow using postgresql database
in other pods in same cluster.
When I try to run some dags that connect to same postgresql with configured
option, the connection to postgresql pods ask for OAUTH authorize
{code:java}
Reading local file:
/root/airflow/logs/anu_funnel_analysis_v2/task_4_get_bigquery_pandas/2018-06-29T02:30:00+00:00/1.log
[2018-11-23 15:56:22,796] {models.py:1335} INFO - Dependencies all met for
<TaskInstance: anu_funnel_analysis_v2.task_4_get_bigquery_pandas
2018-06-29T02:30:00+00:00 [queued]>
[2018-11-23 15:56:22,820] {models.py:1335} INFO - Dependencies all met for
<TaskInstance: anu_funnel_analysis_v2.task_4_get_bigquery_pandas
2018-06-29T02:30:00+00:00 [queued]>
[2018-11-23 15:56:22,821] {models.py:1547} INFO -
--------------------------------------------------------------------------------
Starting attempt 1 of 6
--------------------------------------------------------------------------------
[2018-11-23 15:56:22,838] {models.py:1569} INFO - Executing
<Task(PythonOperator): task_4_get_bigquery_pandas> on 2018-06-29T02:30:00+00:00
[2018-11-23 15:56:22,839] {base_task_runner.py:124} INFO - Running: ['bash',
'-c', u'airflow run anu_funnel_analysis_v2 task_4_get_bigquery_pandas
2018-06-29T02:30:00+00:00 --job_id 82 --raw -sd
DAGS_FOLDER/bigquery_for_funnel.py --cfg_path /tmp/tmpJ1Pe0O']
[2018-11-23 15:56:23,656] {base_task_runner.py:107} INFO - Job 82: Subtask
task_4_get_bigquery_pandas [2018-11-23 15:56:23,655] {settings.py:174} INFO -
setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
[2018-11-23 15:56:23,817] {base_task_runner.py:107} INFO - Job 82: Subtask
task_4_get_bigquery_pandas [2018-11-23 15:56:23,816] {__init__.py:51} INFO -
Using executor SequentialExecutor
[2018-11-23 15:56:23,952] {base_task_runner.py:107} INFO - Job 82: Subtask
task_4_get_bigquery_pandas [2018-11-23 15:56:23,950] {models.py:258} INFO -
Filling up the DagBag from /root/airflow/dags/bigquery_for_funnel.py
[2018-11-23 15:56:24,737] {base_task_runner.py:107} INFO - Job 82: Subtask
task_4_get_bigquery_pandas
/usr/local/lib/python2.7/dist-packages/airflow/contrib/operators/bigquery_operator.py:148:
DeprecationWarning: Deprecated parameter `bql` used in Task id:
task_2_bq_read_new_table_funnel. Use `sql` parameter instead to pass the sql to
be executed. `bql` parameter is deprecated and will be removed in a future
version of Airflow.
[2018-11-23 15:56:24,738] {base_task_runner.py:107} INFO - Job 82: Subtask
task_4_get_bigquery_pandas category=DeprecationWarning)
[2018-11-23 15:56:24,778] {base_task_runner.py:107} INFO - Job 82: Subtask
task_4_get_bigquery_pandas [2018-11-23 15:56:24,777] {cli.py:492} INFO -
Running <TaskInstance: anu_funnel_analysis_v2.task_4_get_bigquery_pandas
2018-06-29T02:30:00+00:00 [running]> on host airflow-5d87576f5b-v5qmg
[2018-11-23 15:56:24,837] {logging_mixin.py:95} INFO - [2018-11-23
15:56:24,837] {base_hook.py:83} INFO - Using connection to:
anudata-postgresql-service
[2018-11-23 15:56:25,335] {logging_mixin.py:95} INFO - Please visit this URL to
authorize this application:
https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=495642085510-k0tmvj2m941jhre2nbqka17vqpjfddtd.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fbigquery&state=zI3Uv6bt3ncdAGIPgWQ5PF36V3uMq0&prompt=consent&access_type=offline
{code}
> Airflow DAGs in Kubernetes ask OAUTH Google Cloud Platform when access
> PostgreSQL database
> ------------------------------------------------------------------------------------------
>
> Key: AIRFLOW-3385
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3385
> Project: Apache Airflow
> Issue Type: Bug
> Components: authentication, kubernetes
> Affects Versions: 1.8.1
> Reporter: RIo
> Priority: Major
>
> I have set and up Airflow in Kubernetes. This Airflow using postgresql
> database in other pods in same cluster.
> When I try to run some dags that connect to same postgresql with configured
> option, the connection to postgresql pods ask for OAUTH authorize
> {code:java}
> Reading local file:
> /root/airflow/logs/anu_funnel_analysis_v2/task_4_get_bigquery_pandas/2018-06-29T02:30:00+00:00/1.log
> [2018-11-23 15:56:22,796] {models.py:1335} INFO - Dependencies all met for
> <TaskInstance: anu_funnel_analysis_v2.task_4_get_bigquery_pandas
> 2018-06-29T02:30:00+00:00 [queued]>
> [2018-11-23 15:56:22,820] {models.py:1335} INFO - Dependencies all met for
> <TaskInstance: anu_funnel_analysis_v2.task_4_get_bigquery_pandas
> 2018-06-29T02:30:00+00:00 [queued]>
> [2018-11-23 15:56:22,821] {models.py:1547} INFO -
> --------------------------------------------------------------------------------
> Starting attempt 1 of 6
> --------------------------------------------------------------------------------
> [2018-11-23 15:56:22,838] {models.py:1569} INFO - Executing
> <Task(PythonOperator): task_4_get_bigquery_pandas> on
> 2018-06-29T02:30:00+00:00
> [2018-11-23 15:56:22,839] {base_task_runner.py:124} INFO - Running: ['bash',
> '-c', u'airflow run anu_funnel_analysis_v2 task_4_get_bigquery_pandas
> 2018-06-29T02:30:00+00:00 --job_id 82 --raw -sd
> DAGS_FOLDER/bigquery_for_funnel.py --cfg_path /tmp/tmpJ1Pe0O']
> [2018-11-23 15:56:23,656] {base_task_runner.py:107} INFO - Job 82: Subtask
> task_4_get_bigquery_pandas [2018-11-23 15:56:23,655] {settings.py:174} INFO -
> setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
> [2018-11-23 15:56:23,817] {base_task_runner.py:107} INFO - Job 82: Subtask
> task_4_get_bigquery_pandas [2018-11-23 15:56:23,816] {__init__.py:51} INFO -
> Using executor SequentialExecutor
> [2018-11-23 15:56:23,952] {base_task_runner.py:107} INFO - Job 82: Subtask
> task_4_get_bigquery_pandas [2018-11-23 15:56:23,950] {models.py:258} INFO -
> Filling up the DagBag from /root/airflow/dags/bigquery_for_funnel.py
> [2018-11-23 15:56:24,737] {base_task_runner.py:107} INFO - Job 82: Subtask
> task_4_get_bigquery_pandas
> /usr/local/lib/python2.7/dist-packages/airflow/contrib/operators/bigquery_operator.py:148:
> DeprecationWarning: Deprecated parameter `bql` used in Task id:
> task_2_bq_read_new_table_funnel. Use `sql` parameter instead to pass the sql
> to be executed. `bql` parameter is deprecated and will be removed in a future
> version of Airflow.
> [2018-11-23 15:56:24,738] {base_task_runner.py:107} INFO - Job 82: Subtask
> task_4_get_bigquery_pandas category=DeprecationWarning)
> [2018-11-23 15:56:24,778] {base_task_runner.py:107} INFO - Job 82: Subtask
> task_4_get_bigquery_pandas [2018-11-23 15:56:24,777] {cli.py:492} INFO -
> Running <TaskInstance: anu_funnel_analysis_v2.task_4_get_bigquery_pandas
> 2018-06-29T02:30:00+00:00 [running]> on host airflow-5d87576f5b-v5qmg
> [2018-11-23 15:56:24,837] {logging_mixin.py:95} INFO - [2018-11-23
> 15:56:24,837] {base_hook.py:83} INFO - Using connection to:
> anudata-postgresql-service
> [2018-11-23 15:56:25,335] {logging_mixin.py:95} INFO - Please visit this URL
> to authorize this application:
> https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=495642085510-k0tmvj2m941jhre2nbqka17vqpjfddtd.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fbigquery&state=zI3Uv6bt3ncdAGIPgWQ5PF36V3uMq0&prompt=consent&access_type=offline
> {code}
>
> Why this happen? in the same pods, one ask for OAUTH and other not?
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
