[jira] [Closed] (AIRFLOW-5215) Add sidecar container support to Pod object
[ https://issues.apache.org/jira/browse/AIRFLOW-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philippe Gagnon closed AIRFLOW-5215. Fix Version/s: 2.0.0 Resolution: Implemented This is no longer required since the Kubernetes Executor now accepts upstream kubernetes `PodV1` objects instead of the custom airflow `Pod` object. > Add sidecar container support to Pod object > --- > > Key: AIRFLOW-5215 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5215 > Project: Apache Airflow > Issue Type: New Feature > Components: scheduler >Affects Versions: 2.0.0 >Reporter: Philippe Gagnon >Assignee: Philippe Gagnon >Priority: Major > Fix For: 2.0.0 > > > Add sidecar container support to Pod object. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5217) Fix Pod docstring
[
https://issues.apache.org/jira/browse/AIRFLOW-5217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philippe Gagnon updated AIRFLOW-5217:
-
Description: {{Pod}} class docstring is currently out of date with regards
to its {{__init__}} method's arguments. (was: {{Pod}} class docstring is
currently out of date with regards to its {{__init__}} method's docstring.)
> Fix Pod docstring
> -
>
> Key: AIRFLOW-5217
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5217
> Project: Apache Airflow
> Issue Type: Improvement
> Components: executors
>Affects Versions: 2.0.0
>Reporter: Philippe Gagnon
>Assignee: Philippe Gagnon
>Priority: Minor
>
> {{Pod}} class docstring is currently out of date with regards to its
> {{__init__}} method's arguments.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (AIRFLOW-5217) Fix Pod docstring
Philippe Gagnon created AIRFLOW-5217:
Summary: Fix Pod docstring
Key: AIRFLOW-5217
URL: https://issues.apache.org/jira/browse/AIRFLOW-5217
Project: Apache Airflow
Issue Type: Improvement
Components: executors
Affects Versions: 2.0.0
Reporter: Philippe Gagnon
Assignee: Philippe Gagnon
{{Pod}} class docstring is currently out of date with regards to its
{{__init__}} method's docstring.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (AIRFLOW-5215) Add sidecar container support to Pod object
Philippe Gagnon created AIRFLOW-5215: Summary: Add sidecar container support to Pod object Key: AIRFLOW-5215 URL: https://issues.apache.org/jira/browse/AIRFLOW-5215 Project: Apache Airflow Issue Type: New Feature Components: scheduler Affects Versions: 2.0.0 Reporter: Philippe Gagnon Assignee: Philippe Gagnon Add sidecar container support to Pod object. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (AIRFLOW-5075) HttpHook cannot handle connections with an empty host field
[
https://issues.apache.org/jira/browse/AIRFLOW-5075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philippe Gagnon updated AIRFLOW-5075:
-
Description:
{{HttpHook}} cannot handle connections with an empty {{host}} field:
{code:java}
Traceback (most recent call last):
File "my_operator.py", line 152, in
- datetime.timedelta(days=1)
File "my_operator.py", line 50, in execute
sess = self.sfdc_hook.get_conn()
File
"/home/ubuntu/.local/lib/python3.6/site-packages/airflow/hooks/http_hook.py",
line 63, in get_conn
if "://" in conn.host:
TypeError: argument of type 'NoneType' is not iterable
{code}
was:
{{HttpHook}} cannot handle connections with an empty {{host}} field:
{code:java}
Traceback (most recent call last):
File "my_operator.py", line 152, in
- datetime.timedelta(days=1)
File "my_operator.py.py", line 50, in execute
sess = self.sfdc_hook.get_conn()
File
"/home/ubuntu/.local/lib/python3.6/site-packages/airflow/hooks/http_hook.py",
line 63, in get_conn
if "://" in conn.host:
TypeError: argument of type 'NoneType' is not iterable
{code}
> HttpHook cannot handle connections with an empty host field
> ---
>
> Key: AIRFLOW-5075
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5075
> Project: Apache Airflow
> Issue Type: Bug
> Components: hooks
>Affects Versions: 1.10.3
>Reporter: Philippe Gagnon
>Assignee: Philippe Gagnon
>Priority: Major
> Fix For: 2.0.0
>
>
> {{HttpHook}} cannot handle connections with an empty {{host}} field:
>
> {code:java}
> Traceback (most recent call last):
> File "my_operator.py", line 152, in
> - datetime.timedelta(days=1)
> File "my_operator.py", line 50, in execute
> sess = self.sfdc_hook.get_conn()
> File
> "/home/ubuntu/.local/lib/python3.6/site-packages/airflow/hooks/http_hook.py",
> line 63, in get_conn
> if "://" in conn.host:
> TypeError: argument of type 'NoneType' is not iterable
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (AIRFLOW-5075) HttpHook cannot handle connections with an empty host field
Philippe Gagnon created AIRFLOW-5075:
Summary: HttpHook cannot handle connections with an empty host
field
Key: AIRFLOW-5075
URL: https://issues.apache.org/jira/browse/AIRFLOW-5075
Project: Apache Airflow
Issue Type: Bug
Components: hooks
Affects Versions: 1.10.3
Reporter: Philippe Gagnon
Assignee: Philippe Gagnon
Fix For: 2.0.0
{{HttpHook}} cannot handle connections with an empty {{host}} field:
{code:java}
Traceback (most recent call last):
File "my_operator.py", line 152, in
- datetime.timedelta(days=1)
File "my_operator.py.py", line 50, in execute
sess = self.sfdc_hook.get_conn()
File
"/home/ubuntu/.local/lib/python3.6/site-packages/airflow/hooks/http_hook.py",
line 63, in get_conn
if "://" in conn.host:
TypeError: argument of type 'NoneType' is not iterable
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (AIRFLOW-4497) KubernetesExecutor fails to launch pods when using run_as_user or fs_group options
[
https://issues.apache.org/jira/browse/AIRFLOW-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884503#comment-16884503
]
Philippe Gagnon commented on AIRFLOW-4497:
--
[~ndigati] Is this solved by AIRFLOW-4516? I think you may be able to close
this issue.
> KubernetesExecutor fails to launch pods when using run_as_user or fs_group
> options
> --
>
> Key: AIRFLOW-4497
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4497
> Project: Apache Airflow
> Issue Type: Bug
> Components: executors
>Affects Versions: 1.10.3
>Reporter: Nick Digati
>Assignee: Philippe Gagnon
>Priority: Minor
> Labels: kubernetes
>
> If you set the run_as_user or the fs_group options in the airflow.cfg file
> when using the KubernetesExecutor the pods will fail to launch with the
> following error message:
>
> {noformat}
> [2019-05-10 05:36:09,207] {pod_launcher.py:60} ERROR - Exception when
> attempting to create Namespaced Pod.
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 57, in run_pod_async
> resp = self._client.create_namespaced_pod(body=req, namespace=pod.namespace)
> File
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 6115, in create_namespaced_pod
> (data) = self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)
> File
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 6206, in create_namespaced_pod_with_http_info
> collection_formats=collection_formats)
> File
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 334, in call_api
> _return_http_data_only, collection_formats, _preload_content,
> _request_timeout)
> File
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 168, in __call_api
> _request_timeout=_request_timeout)
> File
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 377, in request
> body=body)
> File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line
> 266, in POST
> body=body)
> File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line
> 222, in request
> raise ApiException(http_resp=r)
> kubernetes.client.rest.ApiException: (400)
> Reason: Bad Request
> HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json',
> 'Date': 'Fri, 10 May 2019 05:36:09 GMT', 'Content-Length': '428'})
> HTTP response body:
> {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod
> in version \"v1\" cannot be handled as a Pod: v1.Pod.Spec:
> v1.PodSpec.SecurityContext: v1.PodSecurityContext.FSGroup: readUint64:
> unexpected character: \ufffd, error found in #10 byte of ...|sGroup\":
> \"65533\"}}}|..., bigger context ...|, \"affinity\": {}, \"securityContext\":
> {\"fsGroup\": \"65533\"}}}|...","reason":"BadRequest","code":400}{noformat}
>
> It seems like those two options are getting put into the pod manifest as
> strings instead of integers?
>
> My airflow.cfg options:
>
> {noformat}
> [kubernetes]
> run_as_user = 1000
> fs_group = 65533{noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (AIRFLOW-4896) KubernetesExecutorConfig labels default argument is mutable
Philippe Gagnon created AIRFLOW-4896:
Summary: KubernetesExecutorConfig labels default argument is
mutable
Key: AIRFLOW-4896
URL: https://issues.apache.org/jira/browse/AIRFLOW-4896
Project: Apache Airflow
Issue Type: Bug
Components: executors
Affects Versions: 2.0.0
Reporter: Philippe Gagnon
Assignee: Philippe Gagnon
Fix For: 2.0.0
KubernetesExecutorConfig's {{labels}} default argument is mutable (empty dict).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4516) Kubernetes run_as_user and fs_group cannot be strings
[ https://issues.apache.org/jira/browse/AIRFLOW-4516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867937#comment-16867937 ] Philippe Gagnon commented on AIRFLOW-4516: -- Duplicate of AIRFLOW-4497. I am glad you PR'ed this issue though I have been too busy to tackle it so far. > Kubernetes run_as_user and fs_group cannot be strings > - > > Key: AIRFLOW-4516 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4516 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: 1.10.3 >Reporter: Steven Locke >Priority: Minor > Labels: beginner, kubernetes > > The kubernetes operator fails to create pods because the run_as_user and the > fs_group cannot be strings. > Please change `get` to `getint` at > [https://github.com/apache/airflow/blob/master/airflow/executors/kubernetes_executor.py#L160] > and > [https://github.com/apache/airflow/blob/master/airflow/executors/kubernetes_executor.py#L161] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4776) Add type hints to models package
Philippe Gagnon created AIRFLOW-4776: Summary: Add type hints to models package Key: AIRFLOW-4776 URL: https://issues.apache.org/jira/browse/AIRFLOW-4776 Project: Apache Airflow Issue Type: Improvement Components: ci Affects Versions: 1.10.3 Reporter: Philippe Gagnon Assignee: Philippe Gagnon Add type hints to models package -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4762) Test against Python 3.6, 3.7 and 3.8b1
[ https://issues.apache.org/jira/browse/AIRFLOW-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philippe Gagnon updated AIRFLOW-4762: - Description: Airflow is currently tested against python 3.5 only. This may be insufficient as users may wish to run airflow on newer versions, which may introduce breaking changes. (was: Airflow is currently tested against python 3.5 only. This is insufficient as users may wish to run airflow on newer versions, which may introduce breaking changes.) > Test against Python 3.6, 3.7 and 3.8b1 > -- > > Key: AIRFLOW-4762 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4762 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 1.10.3 >Reporter: Philippe Gagnon >Assignee: Philippe Gagnon >Priority: Major > Labels: ci, test > > Airflow is currently tested against python 3.5 only. This may be insufficient > as users may wish to run airflow on newer versions, which may introduce > breaking changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4762) Test against Python 3.6, 3.7 and 3.8b1
[ https://issues.apache.org/jira/browse/AIRFLOW-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philippe Gagnon updated AIRFLOW-4762: - Summary: Test against Python 3.6, 3.7 and 3.8b1 (was: Test against Python 3.7 and 3.8b1) > Test against Python 3.6, 3.7 and 3.8b1 > -- > > Key: AIRFLOW-4762 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4762 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 1.10.3 >Reporter: Philippe Gagnon >Assignee: Philippe Gagnon >Priority: Major > Labels: ci, test > > Airflow is currently tested against python 3.5 only. This is insufficient as > users may wish to run airflow on newer versions, which may introduce breaking > changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4762) Test against Python 3.7 and 3.8b1
[ https://issues.apache.org/jira/browse/AIRFLOW-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philippe Gagnon updated AIRFLOW-4762: - Labels: ci test (was: ) > Test against Python 3.7 and 3.8b1 > - > > Key: AIRFLOW-4762 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4762 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 1.10.3 >Reporter: Philippe Gagnon >Assignee: Philippe Gagnon >Priority: Major > Labels: ci, test > > Airflow is currently tested against python 3.5 only. This is insufficient as > users may wish to run airflow on newer versions, which may introduce breaking > changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-4762) Test against Python 3.7 and 3.8b1
Philippe Gagnon created AIRFLOW-4762: Summary: Test against Python 3.7 and 3.8b1 Key: AIRFLOW-4762 URL: https://issues.apache.org/jira/browse/AIRFLOW-4762 Project: Apache Airflow Issue Type: Improvement Components: ci Affects Versions: 1.10.3 Reporter: Philippe Gagnon Assignee: Philippe Gagnon Airflow is currently tested against python 3.5 only. This is insufficient as users may wish to run airflow on newer versions, which may introduce breaking changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-4497) KubernetesExecutor fails to launch pods when using run_as_user or fs_group options
[
https://issues.apache.org/jira/browse/AIRFLOW-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philippe Gagnon reassigned AIRFLOW-4497:
Assignee: Philippe Gagnon
> KubernetesExecutor fails to launch pods when using run_as_user or fs_group
> options
> --
>
> Key: AIRFLOW-4497
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4497
> Project: Apache Airflow
> Issue Type: Bug
> Components: executors
>Affects Versions: 1.10.3
>Reporter: Nick Digati
>Assignee: Philippe Gagnon
>Priority: Minor
> Labels: kubernetes
>
> If you set the run_as_user or the fs_group options in the airflow.cfg file
> when using the KubernetesExecutor the pods will fail to launch with the
> following error message:
>
> {noformat}
> [2019-05-10 05:36:09,207] {pod_launcher.py:60} ERROR - Exception when
> attempting to create Namespaced Pod.
> Traceback (most recent call last):
> File
> "/usr/local/lib/python3.6/site-packages/airflow/contrib/kubernetes/pod_launcher.py",
> line 57, in run_pod_async
> resp = self._client.create_namespaced_pod(body=req, namespace=pod.namespace)
> File
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 6115, in create_namespaced_pod
> (data) = self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)
> File
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/apis/core_v1_api.py",
> line 6206, in create_namespaced_pod_with_http_info
> collection_formats=collection_formats)
> File
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 334, in call_api
> _return_http_data_only, collection_formats, _preload_content,
> _request_timeout)
> File
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 168, in __call_api
> _request_timeout=_request_timeout)
> File
> "/usr/local/lib/python3.6/site-packages/kubernetes/client/api_client.py",
> line 377, in request
> body=body)
> File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line
> 266, in POST
> body=body)
> File "/usr/local/lib/python3.6/site-packages/kubernetes/client/rest.py", line
> 222, in request
> raise ApiException(http_resp=r)
> kubernetes.client.rest.ApiException: (400)
> Reason: Bad Request
> HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json',
> 'Date': 'Fri, 10 May 2019 05:36:09 GMT', 'Content-Length': '428'})
> HTTP response body:
> {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod
> in version \"v1\" cannot be handled as a Pod: v1.Pod.Spec:
> v1.PodSpec.SecurityContext: v1.PodSecurityContext.FSGroup: readUint64:
> unexpected character: \ufffd, error found in #10 byte of ...|sGroup\":
> \"65533\"}}}|..., bigger context ...|, \"affinity\": {}, \"securityContext\":
> {\"fsGroup\": \"65533\"}}}|...","reason":"BadRequest","code":400}{noformat}
>
> It seems like those two options are getting put into the pod manifest as
> strings instead of integers?
>
> My airflow.cfg options:
>
> {noformat}
> [kubernetes]
> run_as_user = 1000
> fs_group = 65533{noformat}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-4572) Rename prepare_classpath() to prepare_syspath() in settings.py
[
https://issues.apache.org/jira/browse/AIRFLOW-4572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philippe Gagnon closed AIRFLOW-4572.
> Rename prepare_classpath() to prepare_syspath() in settings.py
> --
>
> Key: AIRFLOW-4572
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4572
> Project: Apache Airflow
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.10.3
>Reporter: Philippe Gagnon
>Assignee: Philippe Gagnon
>Priority: Trivial
> Fix For: 1.10.4
>
>
> {{settings.py}} currently defines a {{prepare_classpath()}} function that
> ensures that certain subfolders in {{$AIRFLOW_HOME}} are added to
> {{sys.path}}. The function's naming seems borrowed from Java, and thus could
> be improved by renaming it to {{prepare_syspath()}} which is the equivalent
> Python concept.
> This appears to be an internal API that is only used in {{settings.py}}
> itself, so the change shouldn't have any impact.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-4585) Implement pod mutation hook for KubernetesExecutor and KubernetesPodOperator
[
https://issues.apache.org/jira/browse/AIRFLOW-4585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philippe Gagnon closed AIRFLOW-4585.
> Implement pod mutation hook for KubernetesExecutor and KubernetesPodOperator
>
>
> Key: AIRFLOW-4585
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4585
> Project: Apache Airflow
> Issue Type: New Feature
> Components: executors
>Affects Versions: 1.10.3
>Reporter: Philippe Gagnon
>Assignee: Philippe Gagnon
>Priority: Major
> Labels: executor, kubernetes
> Fix For: 1.10.4
>
>
> This is a proposal to add a mutation hook similar to the {{policy}} function
> in {{airflow_local_settings}} to allow cluster administrators to mutate
> worker pods with more complex behavior than what is currently possible with
> the ini-based configuration.
> This function could be called in {{PodLauncher.run_pod\{_async}}} and apply
> cluster-wide mutations to pods. This could for instance allow users to add
> complex sidecar or init containers to pods launched by their Airflow instance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-4573) airflow_local_settings is imported before sys.path is populated
[
https://issues.apache.org/jira/browse/AIRFLOW-4573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philippe Gagnon closed AIRFLOW-4573.
> airflow_local_settings is imported before sys.path is populated
> ---
>
> Key: AIRFLOW-4573
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4573
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Affects Versions: 1.10.3
>Reporter: Philippe Gagnon
>Assignee: Philippe Gagnon
>Priority: Major
> Labels: settings
> Fix For: 1.10.4
>
>
> {{settings.py}} imports the contents of {{airflow_local_settings.py}} (which
> is expected to live under {{$AIRFLOW_HOME/config}}) before it is added to
> \{{sys.path}}. Thus, unless \{{airflow_local_settings.py}} is added manually
> to {{PYTHONPATH}} before execution, the module will not be found.
> This is caused by the \{{airflow_local_settings}} module getting imported
> before {{prepare_classpath()}} is called, which is the function responsible
> for adding \{{DAGS_FOLDER}}, \{{PLUGINS_FOLDER}} and {{$AIRFLOW_HOME/config}}
> to {{sys.path}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-4585) Implement pod mutation hook for KubernetesExecutor and KubernetesPodOperator
Philippe Gagnon created AIRFLOW-4585:
Summary: Implement pod mutation hook for KubernetesExecutor and
KubernetesPodOperator
Key: AIRFLOW-4585
URL: https://issues.apache.org/jira/browse/AIRFLOW-4585
Project: Apache Airflow
Issue Type: New Feature
Components: executor
Affects Versions: 1.10.3
Reporter: Philippe Gagnon
Assignee: Philippe Gagnon
Fix For: 1.10.4
This is a proposal to add a mutation hook similar to the {{policy}} function in
{{airflow_local_settings}} to allow cluster administrators to mutate worker
pods with more complex behavior than what is currently possible with the
ini-based configuration.
This function could be called in {{PodLauncher.run_pod\{_async}}} and apply
cluster-wide mutations to pods. This could for instance allow users to add
complex sidecar or init containers to pods launched by their Airflow instance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4582) Set KubernetesExecutor worker pod limits and requests based on "resources" task attribute
[
https://issues.apache.org/jira/browse/AIRFLOW-4582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849847#comment-16849847
]
Philippe Gagnon commented on AIRFLOW-4582:
--
@[~dimberman] What do you think? I'll implement this if you are on-board.
> Set KubernetesExecutor worker pod limits and requests based on "resources"
> task attribute
> -
>
> Key: AIRFLOW-4582
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4582
> Project: Apache Airflow
> Issue Type: Improvement
> Components: executor
>Affects Versions: 1.10.3
>Reporter: Philippe Gagnon
>Assignee: Philippe Gagnon
>Priority: Major
> Labels: Kubernetes, executor
> Fix For: 1.10.4
>
>
> At this time the {{KubernetesExecutor}} does not honor resource limits or
> requests set in the {{resources}} task attribute.
> It is possible to set limits and requests by passing an {{executor_config}}
> dict, however this method seems sub-optimal, considering that a generic
> built-in task attribute with essentially the same purpose is already
> available.
> I think we should implement this functionality, but gate it behind an opt-in
> configuration parameter to ensure backwards-compatibility.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-4582) Set KubernetesExecutor worker pod limits and requests based on "resources" task attribute
Philippe Gagnon created AIRFLOW-4582:
Summary: Set KubernetesExecutor worker pod limits and requests
based on "resources" task attribute
Key: AIRFLOW-4582
URL: https://issues.apache.org/jira/browse/AIRFLOW-4582
Project: Apache Airflow
Issue Type: Improvement
Components: executor
Affects Versions: 1.10.3
Reporter: Philippe Gagnon
Assignee: Philippe Gagnon
Fix For: 1.10.4
At this time the {{KubernetesExecutor}} does not honor resource limits or
requests set in the {{resources}} task attribute.
It is possible to set limits and requests by passing an {{executor_config}}
dict, however this method seems sub-optimal, considering that a generic
built-in task attribute with essentially the same purpose is already available.
I think we should implement this functionality, but gate it behind an opt-in
configuration parameter to ensure backwards-compatibility.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4572) Rename prepare_classpath() to prepare_syspath() in settings.py
[
https://issues.apache.org/jira/browse/AIRFLOW-4572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philippe Gagnon updated AIRFLOW-4572:
-
Description:
{{settings.py}} currently defines a {{prepare_classpath()}} function that
ensures that certain subfolders in {{$AIRFLOW_HOME}} are added to {{sys.path}}.
The function's naming seems borrowed from Java, and thus could be improved by
renaming it to {{prepare_syspath()}} which is the equivalent Python concept.
This appears to be an internal API that is only used in {{settings.py}} itself,
so the change shouldn't have any impact.
was:
{{settings.py }}currently defines a {{prepare_classpath()}} function that
ensures that certain subfolders in {{$AIRFLOW_HOME}} are added to {{sys.path}}.
The function's naming seems borrowed from Java, and thus could be improved by
renaming it to {{prepare_syspath()}} which is the equivalent Python concept.
This appears to be an internal API that is only used in {{settings.py}} itself,
so the change shouldn't have any impact.
> Rename prepare_classpath() to prepare_syspath() in settings.py
> --
>
> Key: AIRFLOW-4572
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4572
> Project: Apache Airflow
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.10.3
>Reporter: Philippe Gagnon
>Assignee: Philippe Gagnon
>Priority: Trivial
> Fix For: 1.10.4
>
>
> {{settings.py}} currently defines a {{prepare_classpath()}} function that
> ensures that certain subfolders in {{$AIRFLOW_HOME}} are added to
> {{sys.path}}. The function's naming seems borrowed from Java, and thus could
> be improved by renaming it to {{prepare_syspath()}} which is the equivalent
> Python concept.
> This appears to be an internal API that is only used in {{settings.py}}
> itself, so the change shouldn't have any impact.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4572) Rename prepare_classpath() to prepare_syspath() in settings.py
[
https://issues.apache.org/jira/browse/AIRFLOW-4572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philippe Gagnon updated AIRFLOW-4572:
-
Description:
{{settings.py}}
currently defines a {{prepare_classpath()}} function that ensures that certain
subfolders in {{$AIRFLOW_HOME}} are added to {{sys.path}}. The function's
naming seems borrowed from Java, and thus could be improved by renaming it to
{{prepare_syspath()}} which is the equivalent Python concept.
This appears to be an internal API that is only used in {{settings.py}} itself,
so the change shouldn't have any impact.
was:
{settings.py} currently defines a \{prepare_classpath()} function that ensures
that certain subfolders in \{AIRFLOW_HOME} are added to \{sys.path}. The
function's naming seems borrowed from Java, and thus could be improved by
renaming it to \{prepare_syspath()} which is the equivalent Python concept.
This appears to be an internal API that is only used in \{settings.py} itself,
so the change shouldn't have any impact.
> Rename prepare_classpath() to prepare_syspath() in settings.py
> --
>
> Key: AIRFLOW-4572
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4572
> Project: Apache Airflow
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.10.3
>Reporter: Philippe Gagnon
>Assignee: Philippe Gagnon
>Priority: Trivial
> Fix For: 1.10.4
>
>
> {{settings.py}}
> currently defines a {{prepare_classpath()}} function that ensures that
> certain subfolders in {{$AIRFLOW_HOME}} are added to {{sys.path}}. The
> function's naming seems borrowed from Java, and thus could be improved by
> renaming it to {{prepare_syspath()}} which is the equivalent Python concept.
> This appears to be an internal API that is only used in {{settings.py}}
> itself, so the change shouldn't have any impact.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4572) Rename prepare_classpath() to prepare_syspath() in settings.py
[
https://issues.apache.org/jira/browse/AIRFLOW-4572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philippe Gagnon updated AIRFLOW-4572:
-
Description:
{{settings.py }}currently defines a {{prepare_classpath()}} function that
ensures that certain subfolders in {{$AIRFLOW_HOME}} are added to {{sys.path}}.
The function's naming seems borrowed from Java, and thus could be improved by
renaming it to {{prepare_syspath()}} which is the equivalent Python concept.
This appears to be an internal API that is only used in {{settings.py}} itself,
so the change shouldn't have any impact.
was:
{{settings.py}}
currently defines a {{prepare_classpath()}} function that ensures that certain
subfolders in {{$AIRFLOW_HOME}} are added to {{sys.path}}. The function's
naming seems borrowed from Java, and thus could be improved by renaming it to
{{prepare_syspath()}} which is the equivalent Python concept.
This appears to be an internal API that is only used in {{settings.py}} itself,
so the change shouldn't have any impact.
> Rename prepare_classpath() to prepare_syspath() in settings.py
> --
>
> Key: AIRFLOW-4572
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4572
> Project: Apache Airflow
> Issue Type: Improvement
> Components: core
>Affects Versions: 1.10.3
>Reporter: Philippe Gagnon
>Assignee: Philippe Gagnon
>Priority: Trivial
> Fix For: 1.10.4
>
>
> {{settings.py }}currently defines a {{prepare_classpath()}} function that
> ensures that certain subfolders in {{$AIRFLOW_HOME}} are added to
> {{sys.path}}. The function's naming seems borrowed from Java, and thus could
> be improved by renaming it to {{prepare_syspath()}} which is the equivalent
> Python concept.
> This appears to be an internal API that is only used in {{settings.py}}
> itself, so the change shouldn't have any impact.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4573) airflow_local_settings is imported before sys.path is populated
[
https://issues.apache.org/jira/browse/AIRFLOW-4573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philippe Gagnon updated AIRFLOW-4573:
-
Description:
{{settings.py}} imports the contents of {{airflow_local_settings.py}} (which is
expected to live under {{$AIRFLOW_HOME/config}}) before it is added to
\{{sys.path}}. Thus, unless \{{airflow_local_settings.py}} is added manually to
{{PYTHONPATH}} before execution, the module will not be found.
This is caused by the \{{airflow_local_settings}} module getting imported
before {{prepare_classpath()}} is called, which is the function responsible for
adding \{{DAGS_FOLDER}}, \{{PLUGINS_FOLDER}} and {{$AIRFLOW_HOME/config}} to
{{sys.path}}.
was:
{settings.py} imports the contents of \{airflow_local_settings.py} (which is
expected to live under \{$AIRFLOW_HOME/config}) before it is added to the
\{sys.path}. Thus, unless \{airflow_local_settings.py} is added manually to
\{PYTHONPATH} before execution, the module will not be found.
This is caused by the \{airflow_local_settings} module getting imported before
\{prepare_classpath()} is called, which is the function responsible for adding
\{DAGS_FOLDER}, \{PLUGINS_FOLDER} and \{$AIRFLOW_HOME/config} to \{sys.path}.
> airflow_local_settings is imported before sys.path is populated
> ---
>
> Key: AIRFLOW-4573
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4573
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Affects Versions: 1.10.3
>Reporter: Philippe Gagnon
>Assignee: Philippe Gagnon
>Priority: Major
> Labels: settings
> Fix For: 1.10.4
>
>
> {{settings.py}} imports the contents of {{airflow_local_settings.py}} (which
> is expected to live under {{$AIRFLOW_HOME/config}}) before it is added to
> \{{sys.path}}. Thus, unless \{{airflow_local_settings.py}} is added manually
> to {{PYTHONPATH}} before execution, the module will not be found.
> This is caused by the \{{airflow_local_settings}} module getting imported
> before {{prepare_classpath()}} is called, which is the function responsible
> for adding \{{DAGS_FOLDER}}, \{{PLUGINS_FOLDER}} and {{$AIRFLOW_HOME/config}}
> to {{sys.path}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-4573) airflow_local_settings is imported before sys.path is populated
[
https://issues.apache.org/jira/browse/AIRFLOW-4573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philippe Gagnon updated AIRFLOW-4573:
-
Summary: airflow_local_settings is imported before sys.path is populated
(was: airflow_local_settings is loaded before sys.path is populated)
> airflow_local_settings is imported before sys.path is populated
> ---
>
> Key: AIRFLOW-4573
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4573
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
>Affects Versions: 1.10.3
>Reporter: Philippe Gagnon
>Assignee: Philippe Gagnon
>Priority: Major
> Labels: settings
> Fix For: 1.10.4
>
>
> {settings.py} imports the contents of \{airflow_local_settings.py} (which is
> expected to live under \{$AIRFLOW_HOME/config}) before it is added to the
> \{sys.path}. Thus, unless \{airflow_local_settings.py} is added manually to
> \{PYTHONPATH} before execution, the module will not be found.
> This is caused by the \{airflow_local_settings} module getting imported
> before \{prepare_classpath()} is called, which is the function responsible
> for adding \{DAGS_FOLDER}, \{PLUGINS_FOLDER} and \{$AIRFLOW_HOME/config} to
> \{sys.path}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-4573) airflow_local_settings is loaded before sys.path is populated
Philippe Gagnon created AIRFLOW-4573:
Summary: airflow_local_settings is loaded before sys.path is
populated
Key: AIRFLOW-4573
URL: https://issues.apache.org/jira/browse/AIRFLOW-4573
Project: Apache Airflow
Issue Type: Bug
Components: core
Affects Versions: 1.10.3
Reporter: Philippe Gagnon
Assignee: Philippe Gagnon
Fix For: 1.10.4
{settings.py} imports the contents of \{airflow_local_settings.py} (which is
expected to live under \{$AIRFLOW_HOME/config}) before it is added to the
\{sys.path}. Thus, unless \{airflow_local_settings.py} is added manually to
\{PYTHONPATH} before execution, the module will not be found.
This is caused by the \{airflow_local_settings} module getting imported before
\{prepare_classpath()} is called, which is the function responsible for adding
\{DAGS_FOLDER}, \{PLUGINS_FOLDER} and \{$AIRFLOW_HOME/config} to \{sys.path}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-4572) Rename prepare_classpath() to prepare_syspath() in settings.py
Philippe Gagnon created AIRFLOW-4572:
Summary: Rename prepare_classpath() to prepare_syspath() in
settings.py
Key: AIRFLOW-4572
URL: https://issues.apache.org/jira/browse/AIRFLOW-4572
Project: Apache Airflow
Issue Type: Improvement
Components: core
Affects Versions: 1.10.3
Reporter: Philippe Gagnon
Assignee: Philippe Gagnon
Fix For: 1.10.4
{settings.py} currently defines a \{prepare_classpath()} function that ensures
that certain subfolders in \{AIRFLOW_HOME} are added to \{sys.path}. The
function's naming seems borrowed from Java, and thus could be improved by
renaming it to \{prepare_syspath()} which is the equivalent Python concept.
This appears to be an internal API that is only used in \{settings.py} itself,
so the change shouldn't have any impact.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (AIRFLOW-3274) Add run_as_user and fs_group security context options for KubernetesExecutor
[ https://issues.apache.org/jira/browse/AIRFLOW-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philippe Gagnon closed AIRFLOW-3274. > Add run_as_user and fs_group security context options for KubernetesExecutor > > > Key: AIRFLOW-3274 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3274 > Project: Apache Airflow > Issue Type: Improvement > Components: kubernetes, scheduler >Reporter: Philippe Gagnon >Priority: Major > Fix For: 1.10.3 > > > At this time it is not possible to add `run_as_user` or `fs_group` > securityContext options to worker pods when using KubernetesExecutor. This > makes it harder to use KubernetesExecutor on clusters with pod security > policies which do not allow containers to run as root. > I have already implemented this functionality for my internal use and will > propose a PR soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3274) Add run_as_user and fs_group security context options for KubernetesExecutor
[ https://issues.apache.org/jira/browse/AIRFLOW-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749816#comment-16749816 ] Philippe Gagnon commented on AIRFLOW-3274: -- Yes, sorry. This completely slipped my mind I will prepare this today. > Add run_as_user and fs_group security context options for KubernetesExecutor > > > Key: AIRFLOW-3274 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3274 > Project: Apache Airflow > Issue Type: Improvement > Components: kubernetes, scheduler >Reporter: Philippe Gagnon >Priority: Major > > At this time it is not possible to add `run_as_user` or `fs_group` > securityContext options to worker pods when using KubernetesExecutor. This > makes it harder to use KubernetesExecutor on clusters with pod security > policies which do not allow containers to run as root. > I have already implemented this functionality for my internal use and will > propose a PR soon. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
