[jira] [Commented] (AIRFLOW-3552) Add ImapToS3TransferOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-3552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727995#comment-16727995 ] Felix Uellendall commented on AIRFLOW-3552: --- Sorry. I referenced the wrong id by mistake. > Add ImapToS3TransferOperator > > > Key: AIRFLOW-3552 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3552 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > > This operator transfers mail attachments from a mail server to an amazon s3 > bucket. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3551) Improve BashOperator Test Coverage
[ https://issues.apache.org/jira/browse/AIRFLOW-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727991#comment-16727991 ] Felix Uellendall commented on AIRFLOW-3551: --- You propably mean the `pre_execute` function from the BaseOperator. I mean the `pre_exec` function (see above). > Improve BashOperator Test Coverage > -- > > Key: AIRFLOW-3551 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3551 > Project: Apache Airflow > Issue Type: Test >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Minor > > The current tests for the `BashOperator` are not covering > * pre_exec > * xcom_push_flag -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-3551) Improve BashOperator Test Coverage
[
https://issues.apache.org/jira/browse/AIRFLOW-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727987#comment-16727987
]
Felix Uellendall edited comment on AIRFLOW-3551 at 12/23/18 3:52 PM:
-
I think it is not really testable because it is a inner function and I can not
directly access it.
{code:python}
def pre_exec():
# Restore default signal disposition and invoke setsid
for sig in ('SIGPIPE', 'SIGXFZ', 'SIGXFSZ'):
if hasattr(signal, sig):
signal.signal(getattr(signal, sig), signal.SIG_DFL)
os.setsid()
self.log.info('Running command: %s', self.bash_command)
sub_process = Popen(
['bash', tmp_file.name],
stdout=PIPE,
stderr=STDOUT,
cwd=tmp_dir,
env=self.env,
preexec_fn=pre_exec)
{code}
was (Author: feluelle):
I think it is not really testable because it is a inner function and I can not
directly access it.
{code:python}
def pre_exec():
# Restore default signal disposition and invoke setsid
for sig in ('SIGPIPE', 'SIGXFZ', 'SIGXFSZ'):
if hasattr(signal, sig):
signal.signal(getattr(signal, sig), signal.SIG_DFL)
os.setsid()
self.log.info('Running command: %s', self.bash_command)
sub_process = Popen(
['bash', tmp_file.name],
stdout=PIPE,
stderr=STDOUT,
cwd=tmp_dir,
env=self.env,
preexec_fn=pre_exec)
{code}
If we move it outside we can test it. You think the `pre_exec` function should
be a public function in the `BaseOperator` ?
> Improve BashOperator Test Coverage
> --
>
> Key: AIRFLOW-3551
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3551
> Project: Apache Airflow
> Issue Type: Test
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Minor
>
> The current tests for the `BashOperator` are not covering
> * pre_exec
> * xcom_push_flag
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3551) Improve BashOperator Test Coverage
[
https://issues.apache.org/jira/browse/AIRFLOW-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727987#comment-16727987
]
Felix Uellendall commented on AIRFLOW-3551:
---
I think it is not really testable because it is a inner function and I can not
directly access it.
{code:python}
def pre_exec():
# Restore default signal disposition and invoke setsid
for sig in ('SIGPIPE', 'SIGXFZ', 'SIGXFSZ'):
if hasattr(signal, sig):
signal.signal(getattr(signal, sig), signal.SIG_DFL)
os.setsid()
self.log.info('Running command: %s', self.bash_command)
sub_process = Popen(
['bash', tmp_file.name],
stdout=PIPE,
stderr=STDOUT,
cwd=tmp_dir,
env=self.env,
preexec_fn=pre_exec)
{code}
If we move it outside we can test it. You think the `pre_exec` function should
be a public function in the `BaseOperator` ?
> Improve BashOperator Test Coverage
> --
>
> Key: AIRFLOW-3551
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3551
> Project: Apache Airflow
> Issue Type: Test
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Minor
>
> The current tests for the `BashOperator` are not covering
> * pre_exec
> * xcom_push_flag
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3551) Improve BashOperator Test Coverage
[ https://issues.apache.org/jira/browse/AIRFLOW-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726900#comment-16726900 ] Felix Uellendall commented on AIRFLOW-3551: --- I can't find it in the BaseOperator's implementation. So at the moment it is BashOperator specific. Do you know how I can access the return value of the execute function when the execute function is not explicity getting called like in the test case? > Improve BashOperator Test Coverage > -- > > Key: AIRFLOW-3551 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3551 > Project: Apache Airflow > Issue Type: Test >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Minor > > The current tests for the `BashOperator` are not covering > * pre_exec > * xcom_push_flag -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3554) Remove contrib folder from being omitted by code cov
Felix Uellendall created AIRFLOW-3554: - Summary: Remove contrib folder from being omitted by code cov Key: AIRFLOW-3554 URL: https://issues.apache.org/jira/browse/AIRFLOW-3554 Project: Apache Airflow Issue Type: New Feature Reporter: Felix Uellendall Assignee: Felix Uellendall -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3552) Add ImapToS3TransferOperator
Felix Uellendall created AIRFLOW-3552: - Summary: Add ImapToS3TransferOperator Key: AIRFLOW-3552 URL: https://issues.apache.org/jira/browse/AIRFLOW-3552 Project: Apache Airflow Issue Type: New Feature Reporter: Felix Uellendall Assignee: Felix Uellendall This operator transfers mail attachments from a mail server to an amazon s3 bucket. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3551) Improve BashOperator Test Coverage
Felix Uellendall created AIRFLOW-3551: - Summary: Improve BashOperator Test Coverage Key: AIRFLOW-3551 URL: https://issues.apache.org/jira/browse/AIRFLOW-3551 Project: Apache Airflow Issue Type: Test Reporter: Felix Uellendall Assignee: Felix Uellendall The current tests for the `BashOperator` are not covering * pre_exec * xcom_push_flag -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-2780) Adds IMAP Hook to interact with a mail server
[ https://issues.apache.org/jira/browse/AIRFLOW-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681265#comment-16681265 ] Felix Uellendall edited comment on AIRFLOW-2780 at 11/9/18 11:01 AM: - Ah okay, thank you very much for the clarification :) was (Author: feluelle): Ah okay, thank you very much for the clarification :)) > Adds IMAP Hook to interact with a mail server > - > > Key: AIRFLOW-2780 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2780 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > Fix For: 1.10.1 > > > This Hook connects to a mail server via IMAP to be able to retrieve email > attachments by using [Python's IMAP > Library.|https://docs.python.org/3.6/library/imaplib.html] > Features: > - `has_mail_attachment`: Can be used in a `Sensor` to check if there is an > attachment on the mail server with the given name. > - `retrieve_mail_attachments`: Can be used in an `Operator` to do sth. with > the attachments returned as list of tuple. > - `download_mail_attachments`: Can be used in an `Operator` to download the > attachment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2780) Adds IMAP Hook to interact with a mail server
[ https://issues.apache.org/jira/browse/AIRFLOW-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681265#comment-16681265 ] Felix Uellendall commented on AIRFLOW-2780: --- Ah okay, thank you very much for the clarification :)) > Adds IMAP Hook to interact with a mail server > - > > Key: AIRFLOW-2780 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2780 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > Fix For: 1.10.1 > > > This Hook connects to a mail server via IMAP to be able to retrieve email > attachments by using [Python's IMAP > Library.|https://docs.python.org/3.6/library/imaplib.html] > Features: > - `has_mail_attachment`: Can be used in a `Sensor` to check if there is an > attachment on the mail server with the given name. > - `retrieve_mail_attachments`: Can be used in an `Operator` to do sth. with > the attachments returned as list of tuple. > - `download_mail_attachments`: Can be used in an `Operator` to download the > attachment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3315) Add ImapAttachmentSensor to poke for mail attachments
Felix Uellendall created AIRFLOW-3315: - Summary: Add ImapAttachmentSensor to poke for mail attachments Key: AIRFLOW-3315 URL: https://issues.apache.org/jira/browse/AIRFLOW-3315 Project: Apache Airflow Issue Type: New Feature Reporter: Felix Uellendall Assignee: Felix Uellendall This kind of sensor pokes a mail server for attachments in mails with a given name. It will use the existing [ImapHook|https://issues.apache.org/jira/projects/AIRFLOW/issues/AIRFLOW-2780] to establish a connection to the mail server and search for the attachment in all mails. If an attachment has been found it will immediately stop and return that an attachment has been found for the given name. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2780) Adds IMAP Hook to interact with a mail server
[ https://issues.apache.org/jira/browse/AIRFLOW-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall resolved AIRFLOW-2780. --- Resolution: Done Fix Version/s: 1.10.1 Hey [~ashb] Can I decide in what "Fix Version/s" the hook will be released or who does this? I mean when I pick "1.10.1" here is it sure that it will be in this release? > Adds IMAP Hook to interact with a mail server > - > > Key: AIRFLOW-2780 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2780 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > Fix For: 1.10.1 > > > This Hook connects to a mail server via IMAP to be able to retrieve email > attachments by using [Python's IMAP > Library.|https://docs.python.org/3.6/library/imaplib.html] > Features: > - `has_mail_attachment`: Can be used in a `Sensor` to check if there is an > attachment on the mail server with the given name. > - `retrieve_mail_attachments`: Can be used in an `Operator` to do sth. with > the attachments returned as list of tuple. > - `download_mail_attachments`: Can be used in an `Operator` to download the > attachment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-2824) Disable loading of default connections via airflow config
[
https://issues.apache.org/jira/browse/AIRFLOW-2824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Uellendall reassigned AIRFLOW-2824:
-
Assignee: Andy Cooper
> Disable loading of default connections via airflow config
> -
>
> Key: AIRFLOW-2824
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2824
> Project: Apache Airflow
> Issue Type: Wish
>Reporter: Felix Uellendall
>Assignee: Andy Cooper
>Priority: Major
>
> I would love to have a variable I can set in the airflow.cfg, like the DAG
> examples have, to not load the default connections.
> Either by using {{load_examples}} that is already
> [there|https://github.com/apache/incubator-airflow/blob/dfa7b26ddaca80ee8fd9915ee9f6eac50fac77f6/airflow/config_templates/default_airflow.cfg#L128]
> for loading dag examples or by a new one like {{load_default_connections}}
> to check if the user wants to have it or not.
> The implementation of the default connections starts
> [here|https://github.com/apache/incubator-airflow/blob/9e1d8ee837ea2c23e828d070b6a72a6331d98602/airflow/utils/db.py#L94]
> Let me know what you guys think of it, pls. :)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2780) Adds IMAP Hook to interact with a mail server
[ https://issues.apache.org/jira/browse/AIRFLOW-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-2780: -- Description: This Hook connects to a mail server via IMAP to be able to retrieve email attachments by using [Python's IMAP Library.|https://docs.python.org/3.6/library/imaplib.html] Features: - `has_mail_attachment`: Can be used in a `Sensor` to check if there is an attachment on the mail server with the given name. - `retrieve_mail_attachments`: Can be used in an `Operator` to do sth. with the attachments returned as list of tuple. - `download_mail_attachments`: Can be used in an `Operator` to download the attachment. was: This Hook connects to a mail server via IMAP to be able to retrieve email attachments by using [Python's IMAP Library.|https://docs.python.org/3.6/library/imaplib.html] Features: - `has_mail_attachments`: Can be used in a `Sensor` to check if there is an attachment on the mail server with the given name. - `retrieve_mail_attachments`: Can be used in an `Operator` to do sth. with the attachments returned as list of tuple. - `download_mail_attachments`: Can be used in an `Operator` to download the attachment. Tests: - `test_connect_and_disconnect`: this tests calls the ImapHook as ContextManager - `test_has_mail_attachments_found` - `test_has_mail_attachments_not_found` - `test_has_mail_attachments_with_regex_found` - `test_has_mail_attachments_with_regex_not_found` - `test_retrieve_mail_attachments_found` - `test_retrieve_mail_attachments_not_found` - `test_retrieve_mail_attachments_with_regex_found` - `test_retrieve_mail_attachments_with_regex_not_found` - `test_download_mail_attachments_found` - `test_download_mail_attachments_not_found` - `test_download_mail_attachments_with_regex_found` - `test_download_mail_attachments_with_regex_not_found` > Adds IMAP Hook to interact with a mail server > - > > Key: AIRFLOW-2780 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2780 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > > This Hook connects to a mail server via IMAP to be able to retrieve email > attachments by using [Python's IMAP > Library.|https://docs.python.org/3.6/library/imaplib.html] > Features: > - `has_mail_attachment`: Can be used in a `Sensor` to check if there is an > attachment on the mail server with the given name. > - `retrieve_mail_attachments`: Can be used in an `Operator` to do sth. with > the attachments returned as list of tuple. > - `download_mail_attachments`: Can be used in an `Operator` to download the > attachment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2111) Add the ability to pause/unpause a task within a DAG
[ https://issues.apache.org/jira/browse/AIRFLOW-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599591#comment-16599591 ] Felix Uellendall commented on AIRFLOW-2111: --- I think this violates the design of how Airflow' scheduling is supposed to work, because a task is the smallest entity and not should have a paused state where you have to hold the process it is running on. This would be very complex and I dont think this fits in Airflow's Design Concept. *tl;dr* [~mreid] I think your task is too big when you have this kind of a issue. When you have smaller tasks that *only do one thing* it should be sufficient to pause the dag so after the current running task has succeeded it will be paused and no more tasks will be queued for running. > Add the ability to pause/unpause a task within a DAG > > > Key: AIRFLOW-2111 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2111 > Project: Apache Airflow > Issue Type: New Feature > Components: scheduler, ui >Reporter: Mark Reid >Priority: Minor > > It would be convenient to be able to pause/unpause a task within a DAG > (rather than the entire DAG). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-2934) Pools not respected for internal subdag tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-2934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599585#comment-16599585 ] Felix Uellendall edited comment on AIRFLOW-2934 at 9/1/18 9:26 AM: --- This is a [known bug|https://issues.apache.org/jira/browse/AIRFLOW-41] with SubDag's and LocalExecutor. In Airflow's 1.10 Release there is a quick fix where the SubDagOperator's executor param default value has been set to SequentialExecutor to workaround this issue. But of course if you dont want to upgrade you can set it by [yourself|https://github.com/apache/incubator-airflow/blob/master/airflow/operators/subdag_operator.py#L50] (and FYI https://issues.apache.org/jira/browse/AIRFLOW-74) *EDIT:* Oh and can you try what I described in [this comment|https://issues.apache.org/jira/browse/AIRFLOW-2371?focusedCommentId=16599590=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16599590] was (Author: feluelle): This is a [known bug|https://issues.apache.org/jira/browse/AIRFLOW-41] with SubDag's and LocalExecutor. In Airflow's 1.10 Release there is a quick fix where the SubDagOperator's executor param default value has been set to SequentialExecutor to workaround this issue. But of course if you dont want to upgrade you can set it by [yourself|https://github.com/apache/incubator-airflow/blob/master/airflow/operators/subdag_operator.py#L50] (and FYI https://issues.apache.org/jira/browse/AIRFLOW-74) > Pools not respected for internal subdag tasks > - > > Key: AIRFLOW-2934 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2934 > Project: Apache Airflow > Issue Type: Bug > Components: pools, subdag >Affects Versions: 1.9.0 > Environment: Linux victorvianna 4.14.65-1-MANJARO #1 SMP PREEMPT Sat > Aug 18 13:29:56 UTC 2018 x86_64 GNU/Linux > Python 3.6.6 >Reporter: Victor Vianna >Priority: Blocker > Labels: pool, subdag > Attachments: Screenshot from 2018-08-22 12-32-32.png, dag_pool.py > > > I'm trying to have some subdags execute one task at a time. The way I found > was to create a first pool for the SubdagOperators (pool1 in the attached > code file) and a second one for the internal tasks (pool2). However, it > appears that pools for subdag elements are not being respected. Running > airflow 1.9.0 with LocalExecutor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2371) SubdagOperator ignores pooling
[
https://issues.apache.org/jira/browse/AIRFLOW-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599590#comment-16599590
]
Felix Uellendall commented on AIRFLOW-2371:
---
{noformat}
Using SequentialExecutor is a workaround, however it slows down execution of
subdags.
{noformat}
I think initializing only one SequentialExecutor and passing it to all your
SubDagOperators is quite faster. Did you try that?
> SubdagOperator ignores pooling
> --
>
> Key: AIRFLOW-2371
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2371
> Project: Apache Airflow
> Issue Type: Bug
> Components: subdag
>Affects Versions: Airflow 1.9.0
> Environment: Airflow 1.9.0
>Reporter: Xiao Zhu
>Priority: Major
>
> Basically we need a long term solution for
> https://issues.apache.org/jira/browse/AIRFLOW-74 . Using SequentialExecutor
> is a workaround, however it slows down execution of subdags.
> Looking for a better solution.
> I also see that when running subdags, db connection pools are not shared
> either (since they are separate processes), which causes db disconnection
> issues like https://issues.apache.org/jira/browse/AIRFLOW-2340 on our db
> because there can be too many connections. I hope it can be fixed too when
> subdag operator works with pooling..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-2934) Pools not respected for internal subdag tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-2934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599585#comment-16599585 ] Felix Uellendall edited comment on AIRFLOW-2934 at 9/1/18 9:13 AM: --- This is a [known bug|https://issues.apache.org/jira/browse/AIRFLOW-41] with SubDag's and LocalExecutor. In Airflow's 1.10 Release there is a quick fix where the SubDagOperator's executor param default value has been set to SequentialExecutor to workaround this issue. But of course if you dont want to upgrade you can set it by [yourself|https://github.com/apache/incubator-airflow/blob/master/airflow/operators/subdag_operator.py#L50] (and FYI https://issues.apache.org/jira/browse/AIRFLOW-74) was (Author: feluelle): This is a [known bug|https://issues.apache.org/jira/browse/AIRFLOW-41] with SubDag's and LocalExecutor. In Airflow's 1.10 Release there is a quick fix where the SubDagOperator's executor param default value has been set to SequentialExecutor to workaround this issue. But of course if you dont want to upgrade you can set it by [yourself|https://github.com/apache/incubator-airflow/blob/master/airflow/operators/subdag_operator.py#L50] > Pools not respected for internal subdag tasks > - > > Key: AIRFLOW-2934 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2934 > Project: Apache Airflow > Issue Type: Bug > Components: pools, subdag >Affects Versions: 1.9.0 > Environment: Linux victorvianna 4.14.65-1-MANJARO #1 SMP PREEMPT Sat > Aug 18 13:29:56 UTC 2018 x86_64 GNU/Linux > Python 3.6.6 >Reporter: Victor Vianna >Priority: Blocker > Labels: pool, subdag > Attachments: Screenshot from 2018-08-22 12-32-32.png, dag_pool.py > > > I'm trying to have some subdags execute one task at a time. The way I found > was to create a first pool for the SubdagOperators (pool1 in the attached > code file) and a second one for the internal tasks (pool2). However, it > appears that pools for subdag elements are not being respected. Running > airflow 1.9.0 with LocalExecutor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2934) Pools not respected for internal subdag tasks
[ https://issues.apache.org/jira/browse/AIRFLOW-2934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599585#comment-16599585 ] Felix Uellendall commented on AIRFLOW-2934: --- This is a [known bug|https://issues.apache.org/jira/browse/AIRFLOW-41] with SubDag's and LocalExecutor. In Airflow's 1.10 Release there is a quick fix where the SubDagOperator's executor param default value has been set to SequentialExecutor to workaround this issue. But of course if you dont want to upgrade you can set it by [yourself|https://github.com/apache/incubator-airflow/blob/master/airflow/operators/subdag_operator.py#L50] > Pools not respected for internal subdag tasks > - > > Key: AIRFLOW-2934 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2934 > Project: Apache Airflow > Issue Type: Bug > Components: pools, subdag >Affects Versions: 1.9.0 > Environment: Linux victorvianna 4.14.65-1-MANJARO #1 SMP PREEMPT Sat > Aug 18 13:29:56 UTC 2018 x86_64 GNU/Linux > Python 3.6.6 >Reporter: Victor Vianna >Priority: Blocker > Labels: pool, subdag > Attachments: Screenshot from 2018-08-22 12-32-32.png, dag_pool.py > > > I'm trying to have some subdags execute one task at a time. The way I found > was to create a first pool for the SubdagOperators (pool1 in the attached > code file) and a second one for the internal tasks (pool2). However, it > appears that pools for subdag elements are not being respected. Running > airflow 1.9.0 with LocalExecutor. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2824) Disable loading of default connections via airflow config
[
https://issues.apache.org/jira/browse/AIRFLOW-2824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563690#comment-16563690
]
Felix Uellendall commented on AIRFLOW-2824:
---
Dont know, looks strange to me that it is called upgradedb when it does what
you say.
But I guess I will use it for now but I personally do not like that it is
called upgradedb and it is actually an init without "examples".
A doc patch would not be a fix but an improvement :)
> Disable loading of default connections via airflow config
> -
>
> Key: AIRFLOW-2824
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2824
> Project: Apache Airflow
> Issue Type: Wish
>Reporter: Felix Uellendall
>Priority: Major
>
> I would love to have a variable I can set in the airflow.cfg, like the DAG
> examples have, to not load the default connections.
> Either by using {{load_examples}} that is already
> [there|https://github.com/apache/incubator-airflow/blob/dfa7b26ddaca80ee8fd9915ee9f6eac50fac77f6/airflow/config_templates/default_airflow.cfg#L128]
> for loading dag examples or by a new one like {{load_default_connections}}
> to check if the user wants to have it or not.
> The implementation of the default connections starts
> [here|https://github.com/apache/incubator-airflow/blob/9e1d8ee837ea2c23e828d070b6a72a6331d98602/airflow/utils/db.py#L94]
> Let me know what you guys think of it, pls. :)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2824) Disable loading of default connections via airflow config
[
https://issues.apache.org/jira/browse/AIRFLOW-2824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Uellendall updated AIRFLOW-2824:
--
Issue Type: Wish (was: New Feature)
> Disable loading of default connections via airflow config
> -
>
> Key: AIRFLOW-2824
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2824
> Project: Apache Airflow
> Issue Type: Wish
>Reporter: Felix Uellendall
>Priority: Major
>
> I would love to have a variable I can set in the airflow.cfg, like the DAG
> examples have, to not load the default connections.
> Either by using {{load_examples}} that is already
> [there|https://github.com/apache/incubator-airflow/blob/dfa7b26ddaca80ee8fd9915ee9f6eac50fac77f6/airflow/config_templates/default_airflow.cfg#L128]
> for loading dag examples or by a new one like {{load_default_connections}}
> to check if the user wants to have it or not.
> The implementation of the default connections starts
> [here|https://github.com/apache/incubator-airflow/blob/9e1d8ee837ea2c23e828d070b6a72a6331d98602/airflow/utils/db.py#L94]
> Let me know what you guys think of it, pls. :)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2824) Disable loading of default connections via airflow config
[
https://issues.apache.org/jira/browse/AIRFLOW-2824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Uellendall updated AIRFLOW-2824:
--
Description:
I would love to have a variable I can set in the airflow.cfg, like the DAG
examples have, to not load the default connections.
Either by using {{load_examples}} that is already
[there|https://github.com/apache/incubator-airflow/blob/dfa7b26ddaca80ee8fd9915ee9f6eac50fac77f6/airflow/config_templates/default_airflow.cfg#L128]
for loading dag examples or by a new one like {{load_default_connections}} to
check if the user wants to have it or not.
The implementation of the default connections starts
[here|https://github.com/apache/incubator-airflow/blob/9e1d8ee837ea2c23e828d070b6a72a6331d98602/airflow/utils/db.py#L94]
Let me know what you guys think of it, pls. :)
was:
I would love to have a variable I can set in the airflow.cfg, like the DAG
examples have, to not load the default connections.
Either by using {{load_examples}} that is already there for loading of dag
examples or by a new one like {{load_default_connections}} to check if the user
wants to have it or not.
The implementation of the default connections starts
[here|https://github.com/apache/incubator-airflow/blob/9e1d8ee837ea2c23e828d070b6a72a6331d98602/airflow/utils/db.py#L94]
Let me know what you guys think of it, pls. :)
> Disable loading of default connections via airflow config
> -
>
> Key: AIRFLOW-2824
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2824
> Project: Apache Airflow
> Issue Type: New Feature
>Reporter: Felix Uellendall
>Priority: Major
>
> I would love to have a variable I can set in the airflow.cfg, like the DAG
> examples have, to not load the default connections.
> Either by using {{load_examples}} that is already
> [there|https://github.com/apache/incubator-airflow/blob/dfa7b26ddaca80ee8fd9915ee9f6eac50fac77f6/airflow/config_templates/default_airflow.cfg#L128]
> for loading dag examples or by a new one like {{load_default_connections}}
> to check if the user wants to have it or not.
> The implementation of the default connections starts
> [here|https://github.com/apache/incubator-airflow/blob/9e1d8ee837ea2c23e828d070b6a72a6331d98602/airflow/utils/db.py#L94]
> Let me know what you guys think of it, pls. :)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2824) Disable loading of default connections via airflow config
Felix Uellendall created AIRFLOW-2824:
-
Summary: Disable loading of default connections via airflow config
Key: AIRFLOW-2824
URL: https://issues.apache.org/jira/browse/AIRFLOW-2824
Project: Apache Airflow
Issue Type: New Feature
Reporter: Felix Uellendall
I would love to have a variable I can set in the airflow.cfg, like the DAG
examples have, to not load the default connections.
Either by using {{load_examples}} that is already there for loading of dag
examples or by a new one like {{load_default_connections}} to check if the user
wants to have it or not.
The implementation of the default connections starts
[here|https://github.com/apache/incubator-airflow/blob/9e1d8ee837ea2c23e828d070b6a72a6331d98602/airflow/utils/db.py#L94]
Let me know what you guys think of it, pls. :)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2780) Adds IMAP Hook to interact with a mail server
[ https://issues.apache.org/jira/browse/AIRFLOW-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-2780: -- Summary: Adds IMAP Hook to interact with a mail server (was: Add IMAP Hook to retrieve email attachments from a mail server) > Adds IMAP Hook to interact with a mail server > - > > Key: AIRFLOW-2780 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2780 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > > This Hook connects to a mail server via IMAP to be able to retrieve email > attachments by using [Python's IMAP > Library.|https://docs.python.org/3.6/library/imaplib.html] > Features: > - `has_mail_attachments`: Can be used in a `Sensor` to check if there is an > attachment on the mail server with the given name. > - `retrieve_mail_attachments`: Can be used in an `Operator` to do sth. with > the attachments returned as list of tuple. > - `download_mail_attachments`: Can be used in an `Operator` to download the > attachment. > Tests: > - `test_connect_and_disconnect`: this tests calls the ImapHook as > ContextManager > - `test_has_mail_attachments_found` > - `test_has_mail_attachments_not_found` > - `test_has_mail_attachments_with_regex_found` > - `test_has_mail_attachments_with_regex_not_found` > - `test_retrieve_mail_attachments_found` > - `test_retrieve_mail_attachments_not_found` > - `test_retrieve_mail_attachments_with_regex_found` > - `test_retrieve_mail_attachments_with_regex_not_found` > - `test_download_mail_attachments_found` > - `test_download_mail_attachments_not_found` > - `test_download_mail_attachments_with_regex_found` > - `test_download_mail_attachments_with_regex_not_found` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2780) Add IMAP Hook to retrieve email attachments from a mail server
[
https://issues.apache.org/jira/browse/AIRFLOW-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Uellendall updated AIRFLOW-2780:
--
Description:
This Hook connects to a mail server via IMAP to be able to retrieve email
attachments by using [Python's IMAP
Library.|https://docs.python.org/3.6/library/imaplib.html]
### Description
- `has_mail_attachments`: Can be used in a `Sensor` to check if there is an
attachment on the mail server with the given name.
- `retrieve_mail_attachments`: Can be used in an `Operator` to do sth. with the
attachments returned as list of tuple.
- `download_mail_attachments`: Can be used in an `Operator` to download the
attachment.
### Tests
- `test_connect_and_disconnect`: this tests calls the ImapHook as ContextManager
- `test_has_mail_attachments_found`
- `test_has_mail_attachments_not_found`
- `test_has_mail_attachments_with_regex_found`
- `test_has_mail_attachments_with_regex_not_found`
- `test_retrieve_mail_attachments_found`
- `test_retrieve_mail_attachments_not_found`
- `test_retrieve_mail_attachments_with_regex_found`
- `test_retrieve_mail_attachments_with_regex_not_found`
- `test_download_mail_attachments_found`
- `test_download_mail_attachments_not_found`
- `test_download_mail_attachments_with_regex_found`
- `test_download_mail_attachments_with_regex_not_found`
was:
This Hook connects to a mail server via IMAP to be able to retrieve email
attachments by using [Python's IMAP
Library.|https://docs.python.org/3.6/library/imaplib.html]
Features:
* {{has_mail_attachments}}: Can be used in a {{Sensor}} to check if there is
an attachment on the mail server in a specific mail folder with the given name.
* {{retrieve_mail_attachments}}: Can be used in an {{Operator}} to do sth.
with the attachment after it has been retrieved/downloaded.
> Add IMAP Hook to retrieve email attachments from a mail server
> --
>
> Key: AIRFLOW-2780
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2780
> Project: Apache Airflow
> Issue Type: New Feature
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
>
> This Hook connects to a mail server via IMAP to be able to retrieve email
> attachments by using [Python's IMAP
> Library.|https://docs.python.org/3.6/library/imaplib.html]
> ### Description
> - `has_mail_attachments`: Can be used in a `Sensor` to check if there is an
> attachment on the mail server with the given name.
> - `retrieve_mail_attachments`: Can be used in an `Operator` to do sth. with
> the attachments returned as list of tuple.
> - `download_mail_attachments`: Can be used in an `Operator` to download the
> attachment.
> ### Tests
> - `test_connect_and_disconnect`: this tests calls the ImapHook as
> ContextManager
> - `test_has_mail_attachments_found`
> - `test_has_mail_attachments_not_found`
> - `test_has_mail_attachments_with_regex_found`
> - `test_has_mail_attachments_with_regex_not_found`
> - `test_retrieve_mail_attachments_found`
> - `test_retrieve_mail_attachments_not_found`
> - `test_retrieve_mail_attachments_with_regex_found`
> - `test_retrieve_mail_attachments_with_regex_not_found`
> - `test_download_mail_attachments_found`
> - `test_download_mail_attachments_not_found`
> - `test_download_mail_attachments_with_regex_found`
> - `test_download_mail_attachments_with_regex_not_found`
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2780) Add IMAP Hook to retrieve email attachments from a mail server
[ https://issues.apache.org/jira/browse/AIRFLOW-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Uellendall updated AIRFLOW-2780: -- Description: This Hook connects to a mail server via IMAP to be able to retrieve email attachments by using [Python's IMAP Library.|https://docs.python.org/3.6/library/imaplib.html] Features: - `has_mail_attachments`: Can be used in a `Sensor` to check if there is an attachment on the mail server with the given name. - `retrieve_mail_attachments`: Can be used in an `Operator` to do sth. with the attachments returned as list of tuple. - `download_mail_attachments`: Can be used in an `Operator` to download the attachment. Tests: - `test_connect_and_disconnect`: this tests calls the ImapHook as ContextManager - `test_has_mail_attachments_found` - `test_has_mail_attachments_not_found` - `test_has_mail_attachments_with_regex_found` - `test_has_mail_attachments_with_regex_not_found` - `test_retrieve_mail_attachments_found` - `test_retrieve_mail_attachments_not_found` - `test_retrieve_mail_attachments_with_regex_found` - `test_retrieve_mail_attachments_with_regex_not_found` - `test_download_mail_attachments_found` - `test_download_mail_attachments_not_found` - `test_download_mail_attachments_with_regex_found` - `test_download_mail_attachments_with_regex_not_found` was: This Hook connects to a mail server via IMAP to be able to retrieve email attachments by using [Python's IMAP Library.|https://docs.python.org/3.6/library/imaplib.html] ### Description - `has_mail_attachments`: Can be used in a `Sensor` to check if there is an attachment on the mail server with the given name. - `retrieve_mail_attachments`: Can be used in an `Operator` to do sth. with the attachments returned as list of tuple. - `download_mail_attachments`: Can be used in an `Operator` to download the attachment. ### Tests - `test_connect_and_disconnect`: this tests calls the ImapHook as ContextManager - `test_has_mail_attachments_found` - `test_has_mail_attachments_not_found` - `test_has_mail_attachments_with_regex_found` - `test_has_mail_attachments_with_regex_not_found` - `test_retrieve_mail_attachments_found` - `test_retrieve_mail_attachments_not_found` - `test_retrieve_mail_attachments_with_regex_found` - `test_retrieve_mail_attachments_with_regex_not_found` - `test_download_mail_attachments_found` - `test_download_mail_attachments_not_found` - `test_download_mail_attachments_with_regex_found` - `test_download_mail_attachments_with_regex_not_found` > Add IMAP Hook to retrieve email attachments from a mail server > -- > > Key: AIRFLOW-2780 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2780 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Felix Uellendall >Assignee: Felix Uellendall >Priority: Major > > This Hook connects to a mail server via IMAP to be able to retrieve email > attachments by using [Python's IMAP > Library.|https://docs.python.org/3.6/library/imaplib.html] > Features: > - `has_mail_attachments`: Can be used in a `Sensor` to check if there is an > attachment on the mail server with the given name. > - `retrieve_mail_attachments`: Can be used in an `Operator` to do sth. with > the attachments returned as list of tuple. > - `download_mail_attachments`: Can be used in an `Operator` to download the > attachment. > Tests: > - `test_connect_and_disconnect`: this tests calls the ImapHook as > ContextManager > - `test_has_mail_attachments_found` > - `test_has_mail_attachments_not_found` > - `test_has_mail_attachments_with_regex_found` > - `test_has_mail_attachments_with_regex_not_found` > - `test_retrieve_mail_attachments_found` > - `test_retrieve_mail_attachments_not_found` > - `test_retrieve_mail_attachments_with_regex_found` > - `test_retrieve_mail_attachments_with_regex_not_found` > - `test_download_mail_attachments_found` > - `test_download_mail_attachments_not_found` > - `test_download_mail_attachments_with_regex_found` > - `test_download_mail_attachments_with_regex_not_found` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2780) Add IMAP Hook to retrieve email attachments from a mail server
[
https://issues.apache.org/jira/browse/AIRFLOW-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Uellendall updated AIRFLOW-2780:
--
Description:
This Hook connects to a mail server via IMAP to be able to retrieve email
attachments by using [Python's IMAP
Library.|https://docs.python.org/3.6/library/imaplib.html]
Features:
* {{has_mail_attachments}}: Can be used in a {{Sensor}} to check if there is
an attachment on the mail server in a specific mail folder with the given name.
* {{retrieve_mail_attachments}}: Can be used in an {{Operator}} to do sth.
with the attachment after it has been retrieved/downloaded.
was:
This Hook connects to a mail server via IMAP to be able to retrieve email
attachments by using [Python's IMAP
Library.|https://docs.python.org/3.6/library/imaplib.html]
> Add IMAP Hook to retrieve email attachments from a mail server
> --
>
> Key: AIRFLOW-2780
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2780
> Project: Apache Airflow
> Issue Type: New Feature
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
>
> This Hook connects to a mail server via IMAP to be able to retrieve email
> attachments by using [Python's IMAP
> Library.|https://docs.python.org/3.6/library/imaplib.html]
>
> Features:
> * {{has_mail_attachments}}: Can be used in a {{Sensor}} to check if there is
> an attachment on the mail server in a specific mail folder with the given
> name.
> * {{retrieve_mail_attachments}}: Can be used in an {{Operator}} to do sth.
> with the attachment after it has been retrieved/downloaded.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-2170) The Implement Features section in the CONTRIBUTING.md is incomplete
[
https://issues.apache.org/jira/browse/AIRFLOW-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Uellendall reassigned AIRFLOW-2170:
-
Assignee: Felix Uellendall
> The Implement Features section in the CONTRIBUTING.md is incomplete
> ---
>
> Key: AIRFLOW-2170
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2170
> Project: Apache Airflow
> Issue Type: Improvement
>Affects Versions: 1.9.0
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Trivial
>
> Currently it says:
> {noformat}
> Implement Features
> Look through the Apache Jira for features. Any unassigned "Improvement" issue
> is open to whoever wants to implement it.
> We've created the operators, hooks, macros and executors we needed, but we
> made sure that this part of Airflow is extensible. New operators, hooks and
> operators are very welcomed!{noformat}
> but it would probably be better to change the last sentence to:
> {noformat}
> New operators, hooks, macros and executors are very welcomed!{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2170) The Implement Features section in the CONTRIBUTING.md is incomplete
Felix Uellendall created AIRFLOW-2170:
-
Summary: The Implement Features section in the CONTRIBUTING.md is
incomplete
Key: AIRFLOW-2170
URL: https://issues.apache.org/jira/browse/AIRFLOW-2170
Project: Apache Airflow
Issue Type: Improvement
Affects Versions: 1.9.0
Reporter: Felix Uellendall
Currently it says:
{noformat}
Implement Features
Look through the Apache Jira for features. Any unassigned "Improvement" issue
is open to whoever wants to implement it.
We've created the operators, hooks, macros and executors we needed, but we made
sure that this part of Airflow is extensible. New operators, hooks and
operators are very welcomed!{noformat}
but it would probably be better to change the last sentence to:
{noformat}
New operators, hooks, macros and executors are very welcomed!{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-2135) Errors running setup.py
[
https://issues.apache.org/jira/browse/AIRFLOW-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372041#comment-16372041
]
Felix Uellendall edited comment on AIRFLOW-2135 at 2/21/18 9:31 PM:
Python version 2.7.x or 3.4.x is required for all packages to work.
was (Author: feluelle):
Python version 2.7.x or 3.4.x required for all packages to work.
> Errors running setup.py
> ---
>
> Key: AIRFLOW-2135
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2135
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
> Attachments: psutil_error.png, setproctitle_error.png
>
>
> Hey guys,
>
> i am having trouble setting up the dev enviroment explained
> [here|https://github.com/apache/incubator-airflow/blob/master/CONTRIBUTING.md#development-and-testing]
> I ran the command
> {code:java}
> python setup.py install
> {code}
> and it throws me errors when it tries to install
> {noformat}
> 'setproctitle>=1.1.8, <2'{noformat}
> and
> {noformat}
> 'psutil>=4.2.0, <5.0.0'{noformat}
> I attached both errors.
> I am using fully updated Windows 10 with Python Version 3.6.4. When I ran the
> setup without these packages it runs through. There are no more errors than
> these two.
>
> It would be really great if you could help me with this because I want to
> contribute to the project. :)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AIRFLOW-2135) Errors running setup.py
[
https://issues.apache.org/jira/browse/AIRFLOW-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Uellendall resolved AIRFLOW-2135.
---
Resolution: Not A Bug
Assignee: Felix Uellendall
Python version 2.7.x or 3.4.x required for all packages to work.
> Errors running setup.py
> ---
>
> Key: AIRFLOW-2135
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2135
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Felix Uellendall
>Assignee: Felix Uellendall
>Priority: Major
> Attachments: psutil_error.png, setproctitle_error.png
>
>
> Hey guys,
>
> i am having trouble setting up the dev enviroment explained
> [here|https://github.com/apache/incubator-airflow/blob/master/CONTRIBUTING.md#development-and-testing]
> I ran the command
> {code:java}
> python setup.py install
> {code}
> and it throws me errors when it tries to install
> {noformat}
> 'setproctitle>=1.1.8, <2'{noformat}
> and
> {noformat}
> 'psutil>=4.2.0, <5.0.0'{noformat}
> I attached both errors.
> I am using fully updated Windows 10 with Python Version 3.6.4. When I ran the
> setup without these packages it runs through. There are no more errors than
> these two.
>
> It would be really great if you could help me with this because I want to
> contribute to the project. :)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-2135) Errors running setup.py
[
https://issues.apache.org/jira/browse/AIRFLOW-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372032#comment-16372032
]
Felix Uellendall edited comment on AIRFLOW-2135 at 2/21/18 9:27 PM:
Ok. I figured it out. Airflow based on python > 3.4.x is not working, because
of [setproctitle|https://github.com/dvarrazzo/py-setproctitle/issues/48].
So there is a reason why you put
{noformat}
Please install python(2.7.x or 3.4.x){noformat}
into the
[docs|[https://github.com/apache/incubator-airflow/blob/master/CONTRIBUTING.md#development-and-testing]].
was (Author: feluelle):
Ok. I figured it out. Airflow base on a python version > 3.4.x is not working,
because of
[setproctitle|https://github.com/dvarrazzo/py-setproctitle/issues/48].
So there is a reason why you put
{noformat}
Please install python(2.7.x or 3.4.x){noformat}
into the
[docs|[https://github.com/apache/incubator-airflow/blob/master/CONTRIBUTING.md#development-and-testing]].
> Errors running setup.py
> ---
>
> Key: AIRFLOW-2135
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2135
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Felix Uellendall
>Priority: Major
> Attachments: psutil_error.png, setproctitle_error.png
>
>
> Hey guys,
>
> i am having trouble setting up the dev enviroment explained
> [here|https://github.com/apache/incubator-airflow/blob/master/CONTRIBUTING.md#development-and-testing]
> I ran the command
> {code:java}
> python setup.py install
> {code}
> and it throws me errors when it tries to install
> {noformat}
> 'setproctitle>=1.1.8, <2'{noformat}
> and
> {noformat}
> 'psutil>=4.2.0, <5.0.0'{noformat}
> I attached both errors.
> I am using fully updated Windows 10 with Python Version 3.6.4. When I ran the
> setup without these packages it runs through. There are no more errors than
> these two.
>
> It would be really great if you could help me with this because I want to
> contribute to the project. :)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-2135) Errors running setup.py
[
https://issues.apache.org/jira/browse/AIRFLOW-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372032#comment-16372032
]
Felix Uellendall edited comment on AIRFLOW-2135 at 2/21/18 9:26 PM:
Ok. I figured it out. Airflow base on a python version > 3.4.x is not working,
because of
[setproctitle|https://github.com/dvarrazzo/py-setproctitle/issues/48].
So there is a reason why you put
{noformat}
Please install python(2.7.x or 3.4.x){noformat}
into the
[docs|[https://github.com/apache/incubator-airflow/blob/master/CONTRIBUTING.md#development-and-testing]].
was (Author: feluelle):
Ok. I figured it out. Airflow base on a python version > 3.4.x is not working,
because of
[setproctitle|https://github.com/dvarrazzo/py-setproctitle/issues/48].
So there is a reason why you put
{noformat}
Please install python(2.7.x or 3.4.x){noformat}
into the
[docs|[https://github.com/apache/incubator-airflow/blob/master/CONTRIBUTING.md#development-and-testing]].
> Errors running setup.py
> ---
>
> Key: AIRFLOW-2135
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2135
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Felix Uellendall
>Priority: Major
> Attachments: psutil_error.png, setproctitle_error.png
>
>
> Hey guys,
>
> i am having trouble setting up the dev enviroment explained
> [here|https://github.com/apache/incubator-airflow/blob/master/CONTRIBUTING.md#development-and-testing]
> I ran the command
> {code:java}
> python setup.py install
> {code}
> and it throws me errors when it tries to install
> {noformat}
> 'setproctitle>=1.1.8, <2'{noformat}
> and
> {noformat}
> 'psutil>=4.2.0, <5.0.0'{noformat}
> I attached both errors.
> I am using fully updated Windows 10 with Python Version 3.6.4. When I ran the
> setup without these packages it runs through. There are no more errors than
> these two.
>
> It would be really great if you could help me with this because I want to
> contribute to the project. :)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2135) Errors running setup.py
[
https://issues.apache.org/jira/browse/AIRFLOW-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372032#comment-16372032
]
Felix Uellendall commented on AIRFLOW-2135:
---
Ok. I figured it out. Airflow base on a python version > 3.4.x is not working,
because of
[setproctitle|https://github.com/dvarrazzo/py-setproctitle/issues/48].
So there is a reason why you put
{noformat}
Please install python(2.7.x or 3.4.x){noformat}
into the
[docs|[https://github.com/apache/incubator-airflow/blob/master/CONTRIBUTING.md#development-and-testing].]
> Errors running setup.py
> ---
>
> Key: AIRFLOW-2135
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2135
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Felix Uellendall
>Priority: Major
> Attachments: psutil_error.png, setproctitle_error.png
>
>
> Hey guys,
>
> i am having trouble setting up the dev enviroment explained
> [here|https://github.com/apache/incubator-airflow/blob/master/CONTRIBUTING.md#development-and-testing]
> I ran the command
> {code:java}
> python setup.py install
> {code}
> and it throws me errors when it tries to install
> {noformat}
> 'setproctitle>=1.1.8, <2'{noformat}
> and
> {noformat}
> 'psutil>=4.2.0, <5.0.0'{noformat}
> I attached both errors.
> I am using fully updated Windows 10 with Python Version 3.6.4. When I ran the
> setup without these packages it runs through. There are no more errors than
> these two.
>
> It would be really great if you could help me with this because I want to
> contribute to the project. :)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-2135) Errors running setup.py
[
https://issues.apache.org/jira/browse/AIRFLOW-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16372032#comment-16372032
]
Felix Uellendall edited comment on AIRFLOW-2135 at 2/21/18 9:25 PM:
Ok. I figured it out. Airflow base on a python version > 3.4.x is not working,
because of
[setproctitle|https://github.com/dvarrazzo/py-setproctitle/issues/48].
So there is a reason why you put
{noformat}
Please install python(2.7.x or 3.4.x){noformat}
into the
[docs|[https://github.com/apache/incubator-airflow/blob/master/CONTRIBUTING.md#development-and-testing]].
was (Author: feluelle):
Ok. I figured it out. Airflow base on a python version > 3.4.x is not working,
because of
[setproctitle|https://github.com/dvarrazzo/py-setproctitle/issues/48].
So there is a reason why you put
{noformat}
Please install python(2.7.x or 3.4.x){noformat}
into the
[docs|[https://github.com/apache/incubator-airflow/blob/master/CONTRIBUTING.md#development-and-testing].]
> Errors running setup.py
> ---
>
> Key: AIRFLOW-2135
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2135
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Felix Uellendall
>Priority: Major
> Attachments: psutil_error.png, setproctitle_error.png
>
>
> Hey guys,
>
> i am having trouble setting up the dev enviroment explained
> [here|https://github.com/apache/incubator-airflow/blob/master/CONTRIBUTING.md#development-and-testing]
> I ran the command
> {code:java}
> python setup.py install
> {code}
> and it throws me errors when it tries to install
> {noformat}
> 'setproctitle>=1.1.8, <2'{noformat}
> and
> {noformat}
> 'psutil>=4.2.0, <5.0.0'{noformat}
> I attached both errors.
> I am using fully updated Windows 10 with Python Version 3.6.4. When I ran the
> setup without these packages it runs through. There are no more errors than
> these two.
>
> It would be really great if you could help me with this because I want to
> contribute to the project. :)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
