[
https://issues.apache.org/jira/browse/AIRFLOW-7070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Juraj Šanko updated AIRFLOW-7070:
---------------------------------
Description:
I have noticed this problem with SimpleHttpOperator in Airflow 1.10.9.
When we set it up with conn_id, it does not take into account 'extra'
parameters from connection.
Example:
Airflow -> Admin -> Connection -> Extra:
{"proxies":"https":"http://xxxxxxxx:[email protected]:80"}}
My python skill is very bad, but I think there are 2 things happening:
*1.*
http_hook.py - function *get_conn* - line 78
Context:
{code:java}
if conn.extra:
try:
session.headers.update(conn.extra_dejson)
{code}
This is updating session header, however header can't accept type dictionary,
only string. So first error with this is that the header is incorrectly
formatted (type error).
To me, it makes no sense to use extra field as header source, but if this have
to be kept for compatibility reason, I would suggest excluding python-request
parameters from pushing to headers. Otherwise It may results also in insecure
behavior.
{code:java}
session.proxies.update(extra.get("proxies"))
// This works for me if I disable header code above. (don't use it in
solution){code}
*2.*
**http_hook.py - function *run_and_check* - ** lines 176-184
Extra options here are only taken into consideration from extra_option object,
but they they should be taken from session object as well and overwritten by
connection extra_option object afterwards.
{code:java}
proxies=session.proxies
// replaced this code, as brute force solution.(don't use it in solution){code}
PS: This prohibits us from having sensitive information like proxy login in
secure place therefore creates security concern.
was:
I have noticed this problem with SimpleHttpOperator in Airflow 1.10.9.
When we set it up with conn_id, it does not take into account 'extra'
parameters from connection.
Example:
Airflow -> Admin -> Connection -> Extra:
{"proxies":"https":"http://xxxxxxxx:[email protected]:80"}}
My python skill is very bad, but I think there are 2 things happening:
*1.*
http_hook.py - function *get_conn* - line 78
Context:
{code:java}
if conn.extra:
try:
session.headers.update(conn.extra_dejson)
{code}
This is updating session header, however header can't accept type dictionary,
only string. So first error with this is that the header is incorrectly
formatted (type error).
To me, it makes no sense to use extra field as header source, but if this have
to be kept for compatibility reason, I would suggest excluding python-request
parameters from pushing to headers. Otherwise It may results also in insecure
behavior.
{code:java}
session.proxies.update(extra.get("proxies"))
// This works for me if I disable header code above. (don't use it in
solution){code}
*2.*
**http_hook.py - function *run_and_check* - ** lines 176-184
Extra options here are only taken into consideration from extra_option object,
but they they should be taken from session object as well and overwritten by
connection extra_option object afterwards.
{code:java}
proxies=session.proxies
// replaced this code, as brute force solution.(don't use it in solution){code}
> SimpleHttpOperator does not consider extra attributes from Airflow connection
> -----------------------------------------------------------------------------
>
> Key: AIRFLOW-7070
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7070
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
> Affects Versions: 1.10.9
> Reporter: Juraj Šanko
> Priority: Minor
>
> I have noticed this problem with SimpleHttpOperator in Airflow 1.10.9.
> When we set it up with conn_id, it does not take into account 'extra'
> parameters from connection.
> Example:
> Airflow -> Admin -> Connection -> Extra:
> {"proxies":"https":"http://xxxxxxxx:[email protected]:80"}}
> My python skill is very bad, but I think there are 2 things happening:
> *1.*
> http_hook.py - function *get_conn* - line 78
> Context:
>
> {code:java}
> if conn.extra:
> try:
> session.headers.update(conn.extra_dejson)
> {code}
> This is updating session header, however header can't accept type dictionary,
> only string. So first error with this is that the header is incorrectly
> formatted (type error).
>
> To me, it makes no sense to use extra field as header source, but if this
> have to be kept for compatibility reason, I would suggest excluding
> python-request parameters from pushing to headers. Otherwise It may results
> also in insecure behavior.
>
> {code:java}
> session.proxies.update(extra.get("proxies"))
> // This works for me if I disable header code above. (don't use it in
> solution){code}
>
> *2.*
> **http_hook.py - function *run_and_check* - ** lines 176-184
> Extra options here are only taken into consideration from extra_option
> object, but they they should be taken from session object as well and
> overwritten by connection extra_option object afterwards.
> {code:java}
> proxies=session.proxies
> // replaced this code, as brute force solution.(don't use it in
> solution){code}
>
> PS: This prohibits us from having sensitive information like proxy login in
> secure place therefore creates security concern.
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
