[jira] [Created] (AIRFLOW-2712) Add annotations argument to KubernetesExecutorConfig
Thomas Kunc created AIRFLOW-2712: Summary: Add annotations argument to KubernetesExecutorConfig Key: AIRFLOW-2712 URL: https://issues.apache.org/jira/browse/AIRFLOW-2712 Project: Apache Airflow Issue Type: Improvement Components: executor Affects Versions: Airflow 2.0 Reporter: Thomas Kunc Assignee: Thomas Kunc Fix For: Airflow 2.0 Currently the only annotation available to the KubernetesExecutorConfig is hard-coded. This issue allows annotations to be passed through to that config. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (AIRFLOW-1406) ignore_depends_on_past property ignored by backfill
[
https://issues.apache.org/jira/browse/AIRFLOW-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532179#comment-16532179
]
Dan edited comment on AIRFLOW-1406 at 7/4/18 2:46 AM:
--
good point. I didnt read closely enough here. However, I was actually doing it
correctly using airflow backfill -I and it was still telling me it wasn't
working because it didn't meet the depends_on_past condition. It appears in
this issue's description -I is also tried and it didn't work.
was (Author: dataidi):
good point. I didnt read closely enough here. However, I was actually doing it
correctly using airflow backfill -I and it was still telling me it wasn't
working because it didn't meet the depends_on_past condition.
> ignore_depends_on_past property ignored by backfill
> ---
>
> Key: AIRFLOW-1406
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1406
> Project: Apache Airflow
> Issue Type: Bug
> Components: backfill
>Affects Versions: Airflow 1.8
>Reporter: Tobias Kaymak
>Assignee: Tao Feng
>Priority: Major
>
> Trying to run the following command does not ignore the first depends_on_past
> dependency of the DAG that has depends_on_past set to true:
> {code}airflow backfill -t search_log_sensor -s 2017-07-10 -e 2017-07-11 -I
> google_pipelines{code}
> neither does this one:
> {code}airflow backfill -t search_log_sensor -s 2017-07-10 -e 2017-07-11
> --ignore_depends_on_past google_pipelines {code}
> both result in:
> {code}
> BackfillJob is deadlocked.Some of the deadlocked tasks were unable to run
> because of "depends_on_past" relationships. Try running the backfill with the
> option "ignore_first_depends_on_past=True" or passing "-I" at the command
> line.
> {code}
> Trying to run with
> {code}
> --ignore_depends_on_past=True
> {code}
> yields:
> {code}
> airflow backfill: error: argument -I/--ignore_first_depends_on_past: ignored
> explicit argument 'True'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1406) ignore_depends_on_past property ignored by backfill
[
https://issues.apache.org/jira/browse/AIRFLOW-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532179#comment-16532179
]
Dan commented on AIRFLOW-1406:
--
good point. I didnt read closely enough here. However, I was actually doing it
correctly using airflow backfill -I and it was still telling me it wasn't
working because it didn't meet the depends_on_past condition.
> ignore_depends_on_past property ignored by backfill
> ---
>
> Key: AIRFLOW-1406
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1406
> Project: Apache Airflow
> Issue Type: Bug
> Components: backfill
>Affects Versions: Airflow 1.8
>Reporter: Tobias Kaymak
>Assignee: Tao Feng
>Priority: Major
>
> Trying to run the following command does not ignore the first depends_on_past
> dependency of the DAG that has depends_on_past set to true:
> {code}airflow backfill -t search_log_sensor -s 2017-07-10 -e 2017-07-11 -I
> google_pipelines{code}
> neither does this one:
> {code}airflow backfill -t search_log_sensor -s 2017-07-10 -e 2017-07-11
> --ignore_depends_on_past google_pipelines {code}
> both result in:
> {code}
> BackfillJob is deadlocked.Some of the deadlocked tasks were unable to run
> because of "depends_on_past" relationships. Try running the backfill with the
> option "ignore_first_depends_on_past=True" or passing "-I" at the command
> line.
> {code}
> Trying to run with
> {code}
> --ignore_depends_on_past=True
> {code}
> yields:
> {code}
> airflow backfill: error: argument -I/--ignore_first_depends_on_past: ignored
> explicit argument 'True'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1406) ignore_depends_on_past property ignored by backfill
[
https://issues.apache.org/jira/browse/AIRFLOW-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532047#comment-16532047
]
Tao Feng commented on AIRFLOW-1406:
---
have you tried --ignore_first_depends_on_past
([https://github.com/apache/incubator-airflow/blob/master/airflow/bin/cli.py#L1425)]
option ?
> ignore_depends_on_past property ignored by backfill
> ---
>
> Key: AIRFLOW-1406
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1406
> Project: Apache Airflow
> Issue Type: Bug
> Components: backfill
>Affects Versions: Airflow 1.8
>Reporter: Tobias Kaymak
>Assignee: Tao Feng
>Priority: Major
>
> Trying to run the following command does not ignore the first depends_on_past
> dependency of the DAG that has depends_on_past set to true:
> {code}airflow backfill -t search_log_sensor -s 2017-07-10 -e 2017-07-11 -I
> google_pipelines{code}
> neither does this one:
> {code}airflow backfill -t search_log_sensor -s 2017-07-10 -e 2017-07-11
> --ignore_depends_on_past google_pipelines {code}
> both result in:
> {code}
> BackfillJob is deadlocked.Some of the deadlocked tasks were unable to run
> because of "depends_on_past" relationships. Try running the backfill with the
> option "ignore_first_depends_on_past=True" or passing "-I" at the command
> line.
> {code}
> Trying to run with
> {code}
> --ignore_depends_on_past=True
> {code}
> yields:
> {code}
> airflow backfill: error: argument -I/--ignore_first_depends_on_past: ignored
> explicit argument 'True'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1406) ignore_depends_on_past property ignored by backfill
[
https://issues.apache.org/jira/browse/AIRFLOW-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16532049#comment-16532049
]
Tao Feng commented on AIRFLOW-1406:
---
Based on the souce code read, I don't think --ignore_depends_on_past is
supported by backfill command.
> ignore_depends_on_past property ignored by backfill
> ---
>
> Key: AIRFLOW-1406
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1406
> Project: Apache Airflow
> Issue Type: Bug
> Components: backfill
>Affects Versions: Airflow 1.8
>Reporter: Tobias Kaymak
>Assignee: Tao Feng
>Priority: Major
>
> Trying to run the following command does not ignore the first depends_on_past
> dependency of the DAG that has depends_on_past set to true:
> {code}airflow backfill -t search_log_sensor -s 2017-07-10 -e 2017-07-11 -I
> google_pipelines{code}
> neither does this one:
> {code}airflow backfill -t search_log_sensor -s 2017-07-10 -e 2017-07-11
> --ignore_depends_on_past google_pipelines {code}
> both result in:
> {code}
> BackfillJob is deadlocked.Some of the deadlocked tasks were unable to run
> because of "depends_on_past" relationships. Try running the backfill with the
> option "ignore_first_depends_on_past=True" or passing "-I" at the command
> line.
> {code}
> Trying to run with
> {code}
> --ignore_depends_on_past=True
> {code}
> yields:
> {code}
> airflow backfill: error: argument -I/--ignore_first_depends_on_past: ignored
> explicit argument 'True'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (AIRFLOW-1406) ignore_depends_on_past property ignored by backfill
[
https://issues.apache.org/jira/browse/AIRFLOW-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Feng reassigned AIRFLOW-1406:
-
Assignee: Tao Feng
> ignore_depends_on_past property ignored by backfill
> ---
>
> Key: AIRFLOW-1406
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1406
> Project: Apache Airflow
> Issue Type: Bug
> Components: backfill
>Affects Versions: Airflow 1.8
>Reporter: Tobias Kaymak
>Assignee: Tao Feng
>Priority: Major
>
> Trying to run the following command does not ignore the first depends_on_past
> dependency of the DAG that has depends_on_past set to true:
> {code}airflow backfill -t search_log_sensor -s 2017-07-10 -e 2017-07-11 -I
> google_pipelines{code}
> neither does this one:
> {code}airflow backfill -t search_log_sensor -s 2017-07-10 -e 2017-07-11
> --ignore_depends_on_past google_pipelines {code}
> both result in:
> {code}
> BackfillJob is deadlocked.Some of the deadlocked tasks were unable to run
> because of "depends_on_past" relationships. Try running the backfill with the
> option "ignore_first_depends_on_past=True" or passing "-I" at the command
> line.
> {code}
> Trying to run with
> {code}
> --ignore_depends_on_past=True
> {code}
> yields:
> {code}
> airflow backfill: error: argument -I/--ignore_first_depends_on_past: ignored
> explicit argument 'True'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2711) zendesk hook doesn't handle search endpoint properly
Chris Chow created AIRFLOW-2711: --- Summary: zendesk hook doesn't handle search endpoint properly Key: AIRFLOW-2711 URL: https://issues.apache.org/jira/browse/AIRFLOW-2711 Project: Apache Airflow Issue Type: Bug Affects Versions: 1.9.0 Reporter: Chris Chow the zendesk hook assumes that the api's response includes the expected result in the key with the same name as the api endpoint, e.g. that the results of a query to /api/v2/users.json includes the key 'users'. /api/v2/search.json actually includes results under the key 'results' -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2707) Error accessing log files from web UI
[
https://issues.apache.org/jira/browse/AIRFLOW-2707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531995#comment-16531995
]
Yuri Bendana commented on AIRFLOW-2707:
---
Actually you only need to set the task_log_reader. The rest of it is needed
only if you want to customize logging.
> Error accessing log files from web UI
> -
>
> Key: AIRFLOW-2707
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2707
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.9.0, Airflow 1.9.0
> Environment: Debian
>Reporter: Victor Martin
>Priority: Major
> Attachments: Captura de pantalla 2018-07-02 a las 11.10.33.png
>
>
> After upgrading from Airflow v1.8 to v1.9, local log files are not shown
> anymore on the web UI.
>
> This is the error shown when clicking to "Log" button on a task instance
> details (see attached screenshot):
>
> {code:java}
> Task log handler task does not support read logs. 'NoneType' object has no
> attribute 'read'{code}
>
> I've checked the local filesystem are logs are generated as expected on
> $AIRFLOW_HOME/logs.
>
> Other than configuring celery and postgresql, I'm with default 1.9 airflow.cfg
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2707) Error accessing log files from web UI
[
https://issues.apache.org/jira/browse/AIRFLOW-2707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531951#comment-16531951
]
Yuri Bendana commented on AIRFLOW-2707:
---
I had the same problem in part because I was using the 2.0 (master) version of
the log config file. Copy the
airflow/config_templates/airflow_local_settings.py to AIRFLOW_HOME/config and
add an __init__.py. Then in airflow.cfg set
logging_config_class=airflow_local_settings.DEFAULT_LOGGING_CONFIG and
task_log_reader=file.task.
> Error accessing log files from web UI
> -
>
> Key: AIRFLOW-2707
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2707
> Project: Apache Airflow
> Issue Type: Bug
>Affects Versions: 1.9.0, Airflow 1.9.0
> Environment: Debian
>Reporter: Victor Martin
>Priority: Major
> Attachments: Captura de pantalla 2018-07-02 a las 11.10.33.png
>
>
> After upgrading from Airflow v1.8 to v1.9, local log files are not shown
> anymore on the web UI.
>
> This is the error shown when clicking to "Log" button on a task instance
> details (see attached screenshot):
>
> {code:java}
> Task log handler task does not support read logs. 'NoneType' object has no
> attribute 'read'{code}
>
> I've checked the local filesystem are logs are generated as expected on
> $AIRFLOW_HOME/logs.
>
> Other than configuring celery and postgresql, I'm with default 1.9 airflow.cfg
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-66) Document DAG and Task state cycle
[ https://issues.apache.org/jira/browse/AIRFLOW-66?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531937#comment-16531937 ] Lance Norskog commented on AIRFLOW-66: -- Back when I supported another open source project (Solr) the developers were quite happy to explain the same thing a hundred different times on the mailing list, but were profoundly allergic to making diagrams. -- Lance Norskog lance.nors...@gmail.com Redwood City, CA > Document DAG and Task state cycle > - > > Key: AIRFLOW-66 > URL: https://issues.apache.org/jira/browse/AIRFLOW-66 > Project: Apache Airflow > Issue Type: Improvement > Components: docs >Reporter: Lance Norskog >Priority: Minor > Labels: newbie > > There is no central description of the state sequences for DAGs and Tasks. > Please create a diagram of the state sequences and a definitive text > description of each state. > This type of documentation is useful for beginners. It is also useful as > reference for when the devs discuss subtle changes to the behavior of DAG and > Task execution. > (I just now asked on the mailing list to find out what 'shutdown' state > means, and it was suggested this would be a good newbie JIRA.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2710) Configuration documentation is misleading to the uninitiated
[
https://issues.apache.org/jira/browse/AIRFLOW-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthew Thorley updated AIRFLOW-2710:
-
Description:
The documentation on this page on the configuration page
[https://airflow.apache.org/configuration.html] is slightly misleading
It reads
2. Generate fernet_key, using this code snippet below. fernet_key must be a
base64-encoded 32-byte key.
from cryptography.fernet import Fernet fernet_key= Fernet.generate_key()
print(fernet_key) # your fernet_key, keep it in secured place!
3. Replace {{airflow.cfg}} fernet_key value with the one from step 2.
The value returned in step one is something like
_b'K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs='_
which lead me to believe the config was suppose to be
fernet_key = b'K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs='
When in fact it should be
fernet_key = K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs=
I assumed the config parse needed to know it was a byte string and would handle
the value correctly. After wasting 30mins I was able to figure out the
solution, and probably could have arrived at it sooner had I been more familiar
with python.
But I recommend changing the docs as below to avoid confusion for other new
users.
_from cryptography.fernet import Fernet fernet_key= Fernet.generate_key()
print(fernet_key.decode()) # your fernet_key, keep it in secured place!_
I'll submit a pr for this shortly.
was:
The documentation on this page on the configuration page
[https://airflow.apache.org/configuration.html] is slightly misleading
It say
2. Generate fernet_key, using this code snippet below. fernet_key must be a
base64-encoded 32-byte key.
?? from cryptography.fernet import Fernet fernet_key= Fernet.generate_key()
print(fernet_key) # your fernet_key, keep it in secured place!??
3. Replace {{airflow.cfg}} fernet_key value with the one from step 2.
The value returned in step one is something like
_b'K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs='_
which lead me to believe the config was suppose to be
fernet_key = b'K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs='
When in fact it should be
fernet_key = K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs=
I assumed the config parse needed to know it was a byte string and would handle
the value correctly. After wasting 30mins I was able to figure out the
solution, and probably could have arrived at it sooner had I been more familiar
with python.
But I recommend changing the docs as below to avoid confusion for other new
users.
_from cryptography.fernet import Fernet fernet_key= Fernet.generate_key()
print(fernet_key.decode()) # your fernet_key, keep it in secured place!_
I'll submit a pr for this shortly.
> Configuration documentation is misleading to the uninitiated
>
>
> Key: AIRFLOW-2710
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2710
> Project: Apache Airflow
> Issue Type: Bug
> Components: Documentation
>Reporter: Matthew Thorley
>Priority: Major
>
> The documentation on this page on the configuration page
> [https://airflow.apache.org/configuration.html] is slightly misleading
> It reads
> 2. Generate fernet_key, using this code snippet below. fernet_key must be a
> base64-encoded 32-byte key.
> from cryptography.fernet import Fernet fernet_key= Fernet.generate_key()
> print(fernet_key) # your fernet_key, keep it in secured place!
> 3. Replace {{airflow.cfg}} fernet_key value with the one from step 2.
> The value returned in step one is something like
> _b'K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs='_
> which lead me to believe the config was suppose to be
> fernet_key = b'K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs='
> When in fact it should be
> fernet_key = K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs=
> I assumed the config parse needed to know it was a byte string and would
> handle the value correctly. After wasting 30mins I was able to figure out the
> solution, and probably could have arrived at it sooner had I been more
> familiar with python.
> But I recommend changing the docs as below to avoid confusion for other new
> users.
>
> _from cryptography.fernet import Fernet fernet_key= Fernet.generate_key()
> print(fernet_key.decode()) # your fernet_key, keep it in secured place!_
>
>
> I'll submit a pr for this shortly.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2710) Configuration documentation is misleading to the uninitiated
[
https://issues.apache.org/jira/browse/AIRFLOW-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthew Thorley updated AIRFLOW-2710:
-
Description:
The documentation on this page on the configuration page
[https://airflow.apache.org/configuration.html] is slightly misleading
It say
2. Generate fernet_key, using this code snippet below. fernet_key must be a
base64-encoded 32-byte key.
?? from cryptography.fernet import Fernet fernet_key= Fernet.generate_key()
print(fernet_key) # your fernet_key, keep it in secured place!??
3. Replace {{airflow.cfg}} fernet_key value with the one from step 2.
The value returned in step one is something like
_b'K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs='_
which lead me to believe the config was suppose to be
fernet_key = b'K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs='
When in fact it should be
fernet_key = K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs=
I assumed the config parse needed to know it was a byte string and would handle
the value correctly. After wasting 30mins I was able to figure out the
solution, and probably could have arrived at it sooner had I been more familiar
with python.
But I recommend changing the docs as below to avoid confusion for other new
users.
_from cryptography.fernet import Fernet fernet_key= Fernet.generate_key()
print(fernet_key.decode()) # your fernet_key, keep it in secured place!_
I'll submit a pr for this shortly.
was:
The documentation on this page on the configuration page
[https://airflow.apache.org/configuration.html] is slightly misleading
It says
??2. Generate fernet_key, using this code snippet below. fernet_key must be a
base64-encoded 32-byte key.??
?? from cryptography.fernet import Fernet fernet_key= Fernet.generate_key()
print(fernet_key) # your fernet_key, keep it in secured place!??
?? 3. Replace {{airflow.cfg}} fernet_key value with the one from step 2.??
The value returned in step one is something like
_b'K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs='_
which lead me to believe the config was suppose to be
fernet_key = b'K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs='
When in fact it should be
fernet_key = K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs=
I assumed the config parse needed to know it was a byte string and would handle
the value correctly. After wasting 30mins I was able to figure out the
solution, and probably could have arrived at it sooner had I been more familiar
with python.
But I recommend changing the docs as below to avoid confusion for other new
users.
_from cryptography.fernet import Fernet fernet_key= Fernet.generate_key()
print(fernet_key.decode()) # your fernet_key, keep it in secured place!_
I'll submit a pr for this shortly.
> Configuration documentation is misleading to the uninitiated
>
>
> Key: AIRFLOW-2710
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2710
> Project: Apache Airflow
> Issue Type: Bug
> Components: Documentation
>Reporter: Matthew Thorley
>Priority: Major
>
> The documentation on this page on the configuration page
> [https://airflow.apache.org/configuration.html] is slightly misleading
> It say
> 2. Generate fernet_key, using this code snippet below. fernet_key must be a
> base64-encoded 32-byte key.
> ?? from cryptography.fernet import Fernet fernet_key= Fernet.generate_key()
> print(fernet_key) # your fernet_key, keep it in secured place!??
> 3. Replace {{airflow.cfg}} fernet_key value with the one from step 2.
> The value returned in step one is something like
> _b'K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs='_
> which lead me to believe the config was suppose to be
> fernet_key = b'K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs='
> When in fact it should be
> fernet_key = K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs=
> I assumed the config parse needed to know it was a byte string and would
> handle the value correctly. After wasting 30mins I was able to figure out the
> solution, and probably could have arrived at it sooner had I been more
> familiar with python.
> But I recommend changing the docs as below to avoid confusion for other new
> users.
>
> _from cryptography.fernet import Fernet fernet_key= Fernet.generate_key()
> print(fernet_key.decode()) # your fernet_key, keep it in secured place!_
>
>
> I'll submit a pr for this shortly.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2710) Configuration documentation is misleading to the uninitiated
[
https://issues.apache.org/jira/browse/AIRFLOW-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthew Thorley updated AIRFLOW-2710:
-
Description:
The documentation on this page on the configuration page
[https://airflow.apache.org/configuration.html] is slightly misleading
It says
??2. Generate fernet_key, using this code snippet below. fernet_key must be a
base64-encoded 32-byte key.??
?? from cryptography.fernet import Fernet fernet_key= Fernet.generate_key()
print(fernet_key) # your fernet_key, keep it in secured place!??
?? 3. Replace {{airflow.cfg}} fernet_key value with the one from step 2.??
The value returned in step one is something like
_b'K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs='_
which lead me to believe the config was suppose to be
fernet_key = b'K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs='
When in fact it should be
fernet_key = K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs=
I assumed the config parse needed to know it was a byte string and would handle
the value correctly. After wasting 30mins I was able to figure out the
solution, and probably could have arrived at it sooner had I been more familiar
with python.
But I recommend changing the docs as below to avoid confusion for other new
users.
_from cryptography.fernet import Fernet fernet_key= Fernet.generate_key()
print(fernet_key.decode()) # your fernet_key, keep it in secured place!_
I'll submit a pr for this shortly.
was:
The documentation on this page on the configuration page
[https://airflow.apache.org/configuration.html] is slightly misleading
It says
2. Generate fernet_key, using this code snippet below. fernet_key must be a
base64-encoded 32-byte key.
from cryptography.fernet import Fernet fernet_key= Fernet.generate_key()
print(fernet_key) # your fernet_key, keep it in secured place!
3. Replace {{airflow.cfg}} fernet_key value with the one from step 2.
The value returned in step one is something like
b'K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs='
which lead me to believe the config was suppose to be
fernet_key = b'K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs='
When in fact it should be
fernet_key = K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs=
I assumed the config parse needed to know it was a byte string and would handle
the value correctly. After wasting 30mins I was able to figure out the
solution, and probably could have arrived at it sooner had I been more familiar
with python.
But I recommend changing the docs as below to avoid confusion for other new
users.
from cryptography.fernet import Fernet fernet_key= Fernet.generate_key()
print(fernet_key.decode()) # your fernet_key, keep it in secured place!
I'll submit a pr for this shortly.
> Configuration documentation is misleading to the uninitiated
>
>
> Key: AIRFLOW-2710
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2710
> Project: Apache Airflow
> Issue Type: Bug
> Components: Documentation
>Reporter: Matthew Thorley
>Priority: Major
>
> The documentation on this page on the configuration page
> [https://airflow.apache.org/configuration.html] is slightly misleading
> It says
> ??2. Generate fernet_key, using this code snippet below. fernet_key must be a
> base64-encoded 32-byte key.??
> ?? from cryptography.fernet import Fernet fernet_key= Fernet.generate_key()
> print(fernet_key) # your fernet_key, keep it in secured place!??
> ?? 3. Replace {{airflow.cfg}} fernet_key value with the one from step 2.??
> The value returned in step one is something like
> _b'K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs='_
> which lead me to believe the config was suppose to be
> fernet_key = b'K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs='
> When in fact it should be
> fernet_key = K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs=
> I assumed the config parse needed to know it was a byte string and would
> handle the value correctly. After wasting 30mins I was able to figure out the
> solution, and probably could have arrived at it sooner had I been more
> familiar with python.
> But I recommend changing the docs as below to avoid confusion for other new
> users.
>
> _from cryptography.fernet import Fernet fernet_key= Fernet.generate_key()
> print(fernet_key.decode()) # your fernet_key, keep it in secured place!_
>
>
> I'll submit a pr for this shortly.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2710) Configuration documentation is misleading to the uninitiated
[
https://issues.apache.org/jira/browse/AIRFLOW-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531865#comment-16531865
]
Matthew Thorley commented on AIRFLOW-2710:
--
Link to pull request https://github.com/apache/incubator-airflow/pull/3574
> Configuration documentation is misleading to the uninitiated
>
>
> Key: AIRFLOW-2710
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2710
> Project: Apache Airflow
> Issue Type: Bug
> Components: Documentation
>Reporter: Matthew Thorley
>Priority: Major
>
> The documentation on this page on the configuration page
> [https://airflow.apache.org/configuration.html] is slightly misleading
> It says
> 2. Generate fernet_key, using this code snippet below. fernet_key must be a
> base64-encoded 32-byte key.
> from cryptography.fernet import Fernet fernet_key= Fernet.generate_key()
> print(fernet_key) # your fernet_key, keep it in secured place!
> 3. Replace {{airflow.cfg}} fernet_key value with the one from step 2.
> The value returned in step one is something like
> b'K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs='
> which lead me to believe the config was suppose to be
> fernet_key = b'K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs='
> When in fact it should be
> fernet_key = K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs=
> I assumed the config parse needed to know it was a byte string and would
> handle the value correctly. After wasting 30mins I was able to figure out the
> solution, and probably could have arrived at it sooner had I been more
> familiar with python.
> But I recommend changing the docs as below to avoid confusion for other new
> users.
>
> from cryptography.fernet import Fernet fernet_key= Fernet.generate_key()
> print(fernet_key.decode()) # your fernet_key, keep it in secured place!
>
>
> I'll submit a pr for this shortly.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-1406) ignore_depends_on_past property ignored by backfill
[
https://issues.apache.org/jira/browse/AIRFLOW-1406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531863#comment-16531863
]
Dan commented on AIRFLOW-1406:
--
+1 I am running into the same thing.
> ignore_depends_on_past property ignored by backfill
> ---
>
> Key: AIRFLOW-1406
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1406
> Project: Apache Airflow
> Issue Type: Bug
> Components: backfill
>Affects Versions: Airflow 1.8
>Reporter: Tobias Kaymak
>Priority: Major
>
> Trying to run the following command does not ignore the first depends_on_past
> dependency of the DAG that has depends_on_past set to true:
> {code}airflow backfill -t search_log_sensor -s 2017-07-10 -e 2017-07-11 -I
> google_pipelines{code}
> neither does this one:
> {code}airflow backfill -t search_log_sensor -s 2017-07-10 -e 2017-07-11
> --ignore_depends_on_past google_pipelines {code}
> both result in:
> {code}
> BackfillJob is deadlocked.Some of the deadlocked tasks were unable to run
> because of "depends_on_past" relationships. Try running the backfill with the
> option "ignore_first_depends_on_past=True" or passing "-I" at the command
> line.
> {code}
> Trying to run with
> {code}
> --ignore_depends_on_past=True
> {code}
> yields:
> {code}
> airflow backfill: error: argument -I/--ignore_first_depends_on_past: ignored
> explicit argument 'True'
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-2710) Configuration documentation is misleading to the uninitiated
Matthew Thorley created AIRFLOW-2710:
Summary: Configuration documentation is misleading to the
uninitiated
Key: AIRFLOW-2710
URL: https://issues.apache.org/jira/browse/AIRFLOW-2710
Project: Apache Airflow
Issue Type: Bug
Components: Documentation
Reporter: Matthew Thorley
The documentation on this page on the configuration page
[https://airflow.apache.org/configuration.html] is slightly misleading
It says
2. Generate fernet_key, using this code snippet below. fernet_key must be a
base64-encoded 32-byte key.
from cryptography.fernet import Fernet fernet_key= Fernet.generate_key()
print(fernet_key) # your fernet_key, keep it in secured place!
3. Replace {{airflow.cfg}} fernet_key value with the one from step 2.
The value returned in step one is something like
b'K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs='
which lead me to believe the config was suppose to be
fernet_key = b'K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs='
When in fact it should be
fernet_key = K_8Yv52REP1qsa7OPupKYJe_CzngMI_KqwfM-2qAyVs=
I assumed the config parse needed to know it was a byte string and would handle
the value correctly. After wasting 30mins I was able to figure out the
solution, and probably could have arrived at it sooner had I been more familiar
with python.
But I recommend changing the docs as below to avoid confusion for other new
users.
from cryptography.fernet import Fernet fernet_key= Fernet.generate_key()
print(fernet_key.decode()) # your fernet_key, keep it in secured place!
I'll submit a pr for this shortly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-66) Document DAG and Task state cycle
[ https://issues.apache.org/jira/browse/AIRFLOW-66?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531840#comment-16531840 ] Tim Swast commented on AIRFLOW-66: -- This would probably be appropriate for the Tasks concepts documentation at https://airflow.incubator.apache.org/concepts.html#tasks > Document DAG and Task state cycle > - > > Key: AIRFLOW-66 > URL: https://issues.apache.org/jira/browse/AIRFLOW-66 > Project: Apache Airflow > Issue Type: Improvement > Components: docs >Reporter: Lance Norskog >Priority: Minor > Labels: newbie > > There is no central description of the state sequences for DAGs and Tasks. > Please create a diagram of the state sequences and a definitive text > description of each state. > This type of documentation is useful for beginners. It is also useful as > reference for when the devs discuss subtle changes to the behavior of DAG and > Task execution. > (I just now asked on the mailing list to find out what 'shutdown' state > means, and it was suggested this would be a good newbie JIRA.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2099) Task details cannot be shown when PythonOperator calls partial function / class instance with __call__
[
https://issues.apache.org/jira/browse/AIRFLOW-2099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531555#comment-16531555
]
Matthew Revell commented on AIRFLOW-2099:
-
PR was closed due to inactivity. New PR created with necessary changes:
https://github.com/apache/incubator-airflow/pull/3571
> Task details cannot be shown when PythonOperator calls partial function /
> class instance with __call__
> --
>
> Key: AIRFLOW-2099
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2099
> Project: Apache Airflow
> Issue Type: Bug
> Components: ui
>Affects Versions: Airflow 1.8
>Reporter: Matthew Revell
>Assignee: Matthew Revell
>Priority: Minor
>
> There are several scenarios where the inspect.getsource() method fails with:
> {{object at 0x> is not a module, class, method, function, traceback,
> frame, or code object}}
> One such scenario is described in
> [AIRFLOW-1027|https://issues.apache.org/jira/browse/AIRFLOW-1027] where a
> partial function is used. Another is when an instance of a class which
> implements __call__() is used.
> Example:
> {{class MyClass(object):}}
> {{ def __init__(self):}}
> {{ pass}}
> {{ def __call__(self):}}
> {{ pass}}
> {{my_class = MyClass()}}
> {{dag_task = PythonOperator(}}
> {{ task_id='dag_task',}}
> {{ dag=dag, }}
> {{ python_callable=my_class,}}
> {{)}}
> There exists a PR for AIRFLOW-1027, however, this fix does not address this
> other scenario, and also does not guard against any other edge cases which my
> result in this error in future.
> A better solution would be to catch known scenarios with work arounds, and
> default to reporting that the source is unavailable for unknown cases. This
> would at least display the Task Instance details in every case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2064) Polish timezone implementation
[ https://issues.apache.org/jira/browse/AIRFLOW-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530921#comment-16530921 ] Manu Zhang commented on AIRFLOW-2064: - Hi, I find that "time"s on the dashboard are still UTC even if a local timezone is configured. For example, the clock [https://github.com/apache/incubator-airflow/blob/master/airflow/www/templates/admin/master.html#L35] is hardcoded to "UTCseconds". Also, the "Last Run" time, started time, etc of a Dag are in UTC time. > Polish timezone implementation > -- > > Key: AIRFLOW-2064 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2064 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Bolke de Bruin >Priority: Blocker > Fix For: 1.10.0 > > > Couple of things are left over after moving to time zone support: > > # End_dates within dags should be converted to UTC by using the time zone of > start_date if naive > # Task instances that are instantiated without timezone information for > their execution_date should convert those to UTC by using the DAG's timezone > or configured > # Some doc polishing > # Tests should be added that cover more of the edge cases -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2709) Improve error handling in Databricks hook
[ https://issues.apache.org/jira/browse/AIRFLOW-2709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victor Jimenez updated AIRFLOW-2709: Description: The Databricks hook handles both connection and timeout errors. However, Databricks sometimes returns a temporarily unavailable error. That error is neither a connection nor timeout one. It is just an HTTPError containing the following text in the response: TEMPORARILY_UNAVAILABLE. The current error handling in the hook should be enhanced to support this error. Also, the Databricks hook contains retry logic. Yet, it does not support sleeping for some time between requests. This creates a problem in handling errors such as the TEMPORARILY_UNAVAILABLE one. This error typically resolves after a few seconds. Adding support for sleeping between retry attempts would really help to enhance the reliability of Databricks operations. was: Databricks hook handles both connection and timeout errors. However, Databricks sometimes returns a temporarily unavailable error. That error is neither a connection nor timeout one. It is just an HTTPError with the Databricks hook contains retry logic. Yet, it does not support sleeping for some time between requests. This creates a problem in some circumstances when Databricks returns This error typically resolves after a few seconds. Adding support for sleeping between retry attempts would really help to enhance the reliability of Databricks operations. > Improve error handling in Databricks hook > - > > Key: AIRFLOW-2709 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2709 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib, hooks >Reporter: Victor Jimenez >Priority: Major > > The Databricks hook handles both connection and timeout errors. However, > Databricks sometimes returns a temporarily unavailable error. That error is > neither a connection nor timeout one. It is just an HTTPError containing the > following text in the response: TEMPORARILY_UNAVAILABLE. The current error > handling in the hook should be enhanced to support this error. > Also, the Databricks hook contains retry logic. Yet, it does not support > sleeping for some time between requests. This creates a problem in handling > errors such as the TEMPORARILY_UNAVAILABLE one. This error typically resolves > after a few seconds. Adding support for sleeping between retry attempts would > really help to enhance the reliability of Databricks operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2709) Improve error handling in Databricks hook
[ https://issues.apache.org/jira/browse/AIRFLOW-2709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victor Jimenez updated AIRFLOW-2709: Description: Databricks hook handles both connection and timeout errors. However, Databricks sometimes returns a temporarily unavailable error. That error is neither a connection nor timeout one. It is just an HTTPError with the Databricks hook contains retry logic. Yet, it does not support sleeping for some time between requests. This creates a problem in some circumstances when Databricks returns This error typically resolves after a few seconds. Adding support for sleeping between retry attempts would really help to enhance the reliability of Databricks operations. was: Databricks hook contains retry logic. Yet, it does not support sleeping for some time between requests. This creates a problem in some circumstances when Databricks returns a temporarily unavailable error. This error typically resolves after a few seconds. Adding support for sleeping between retry attempts would really help to enhance the reliability of Databricks operations. > Improve error handling in Databricks hook > - > > Key: AIRFLOW-2709 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2709 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib, hooks >Reporter: Victor Jimenez >Priority: Major > > Databricks hook handles both connection and timeout errors. However, > Databricks sometimes returns a temporarily unavailable error. That error is > neither a connection nor timeout one. It is just an HTTPError with the > > Databricks hook contains retry logic. Yet, it does not support sleeping for > some time between requests. This creates a problem in some circumstances when > Databricks returns > This error typically resolves after a few seconds. Adding support for > sleeping between retry attempts would really help to enhance the reliability > of Databricks operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2709) Improve error handling in Databricks hook
[ https://issues.apache.org/jira/browse/AIRFLOW-2709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victor Jimenez updated AIRFLOW-2709: Summary: Improve error handling in Databricks hook (was: Add sleep time to retry logic in Databricks hook) > Improve error handling in Databricks hook > - > > Key: AIRFLOW-2709 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2709 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib, hooks >Reporter: Victor Jimenez >Priority: Major > > Databricks hook contains retry logic. Yet, it does not support sleeping for > some time between requests. This creates a problem in some circumstances when > Databricks returns a temporarily unavailable error. > This error typically resolves after a few seconds. Adding support for > sleeping between retry attempts would really help to enhance the reliability > of Databricks operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
