[GitHub] [airflow] dstandish edited a comment on issue #6210: [AIRFLOW-5567] BaseReschedulePokeOperator
dstandish edited a comment on issue #6210: [AIRFLOW-5567] BaseReschedulePokeOperator URL: https://github.com/apache/airflow/pull/6210#issuecomment-558030520 > Having a state table will have a fundamental impact on the idempotency of the execution of the tasks. It's optional to use such a thing. Just like it is with XCom. If you don't use it, nothing is changed. > Why would the manual triggering of a dag introduce issues, the execution date will be equal to the moment that it was triggered. I think it should work as well. Because execution_date is run date minus one interval, and `xcom_pull` sorts by execution_date. So, suppose I want to persist state with XCom (which I do in many jobs), and I have a daily job, running at midnight. At end of each run, we push some value to XCom. At start of next job, we retrieve last value and use it somehow. Consider this case: * run 1: 12am D1 * run 2: manually triggered at 8am (exec date is D1 8am; xcom retrieves from run 1) * run 3: 12am D2 * run 4: 12am D3 * run 5: 12am D4 Outcome: * Run 3 will retrive the XCom from run 1, because its execution date is prior to run 2 execution date. * Run 4 retrieves run 2 for same reason. * Run 5 retrieves run 4 (finally things are back in order); run 3 xcom is never retrieved by any job. The schedule interval edge PR would resolve the execution date ordering problem. But if XCom is cleared at start of task, it is remains problematic as a mechanism for state persistence. > Since this will introduce such as a fundamental change to the way operators were intended, being idempotent, I think it would be great to first start an AIP on the topic, so we can have a clear and structured approach. An AIP sounds appropriate. I am a bit skeptical of the notion that this is a radical change; I would be shocked if stateful processes were not already an extremely common use pattern. I suspect the effect of such a change would be more to provide better support for an existing use pattern than to change the way airflow is used (because it can already be done with XCom, if only imperfectly). Anyway, I have occasionally rambled on dev list about these issues. I am not sure what the best solution is. I wish there could be clearer and more generalized separation between the concepts of "run date" and "interval of interest", but I am not sure what that should look like. But having a simple way to persist state would be of great immediate help to me, and to this PR, incidentally. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dstandish edited a comment on issue #6210: [AIRFLOW-5567] BaseReschedulePokeOperator
dstandish edited a comment on issue #6210: [AIRFLOW-5567] BaseReschedulePokeOperator URL: https://github.com/apache/airflow/pull/6210#issuecomment-558030520 > Having a state table will have a fundamental impact on the idempotency of the execution of the tasks. It's optional to use such a thing. Just like it is with XCom. If you don't use it, nothing is changed. > Why would the manual triggering of a dag introduce issues, the execution date will be equal to the moment that it was triggered. I think it should work as well. Because execution_date is run date minus one interval, and `xcom_pull` sorts by execution_date. So, suppose I want to persist state with XCom (which I do in many jobs), and I have a daily job, running at midnight. At end of each run, we push some value to XCom. At start of next job, we retrieve last value and use it somehow. Consider this case: * run 1: 12am D1 * run 2: manually triggered at 8am (exec date is D1 8am; xcom retrieves from run 1) * run 3: 12am D2 * run 4: 12am D3 * run 5: 12am D4 Outcome: * Run 3 will retrive the XCom from run 1, because its execution date is prior to run 2 execution date. * Run 4 retrieves run 2 for same reason. * Run 5 retrieves run 4 (finally things are back in order); run 3 xcom is never retrieved by any job. The schedule interval edge PR would resolve the execution date ordering problem. But if XCom is cleared at start of task, it is remains problematic as a mechanism for state persistence. > Since this will introduce such as a fundamental change to the way operators were intended, being idempotent, I think it would be great to first start an AIP on the topic, so we can have a clear and structured approach. An AIP sounds appropriate. I am just a bit skeptical of the notion that this is some radical change; I would be shocked if stateful processes were not already an extremely common use pattern. I suspect the effect of such a change would be more to provide better support for an existing use pattern than to change the way airflow is used (because it can already be done with XCom, if only imperfectly). Airflow provides great support for a particular kind of "idempotent" task, but surely it doesn't say this is the only way it should be used! Anyway, I have occasionally rambled on dev list about these issues. I am not sure what the best solution is. I wish there could be clearer and more generalized separation between the concepts of "run date" and "interval of interest", but I am not sure what that should look like. But having a simple way to persist state would be of great immediate help to me, and to this PR, incidentally. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dstandish edited a comment on issue #6210: [AIRFLOW-5567] BaseReschedulePokeOperator
dstandish edited a comment on issue #6210: [AIRFLOW-5567] BaseReschedulePokeOperator URL: https://github.com/apache/airflow/pull/6210#issuecomment-558030520 > Having a state table will have a fundamental impact on the idempotency of the execution of the tasks. It's optional to use such a thing. Just like it is with XCom. If you don't use it, nothing is changed. > Why would the manual triggering of a dag introduce issues, the execution date will be equal to the moment that it was triggered. I think it should work as well. Because execution_date is run date minus one interval, and `xcom_pull` sorts by execution_date. So, suppose I want to persist state with XCom (which I do in many jobs), and I have a daily job, running at midnight. At end of each run, we push some value to XCom. At start of next job, we retrieve last value and use it somehow. Consider this case: * run 1: 12am D1 * run 2: manually triggered at 8am (exec date is D1 8am; xcom retrieves from run 1) * run 3: 12am D2 * run 4: 12am D3 * run 5: 12am D4 Outcome: * Run 3 will retrive the XCom from run 1, because its execution date is prior to run 2 execution date. * Run 4 retrieves run 2 for same reason. * Run 5 retrieves run 4 (finally things are back in order); run 3 xcom is never retrieved by any job. The schedule interval edge PR would resolve the execution date ordering problem. But if XCom is cleared at start of task, it is remains problematic as a mechanism for state persistence. > Since this will introduce such as a fundamental change to the way operators were intended, being idempotent, I think it would be great to first start an AIP on the topic, so we can have a clear and structured approach. An AIP sounds appropriate. I am just a bit skeptical of the notion that this is some radical change; I would be shocked if stateful processes were not already an extremely common use pattern. Here the goal would be to provide better support for them out of the box. Airflow provides great support for a particular kind of "idempotent" task, but surely it doesn't say this is the only way it should be used! Anyway, I have occasionally rambled on dev list about these issues. I am not sure what the best solution is. I wish there could be clearer and more generalized separation between the concepts of "run date" and "interval of interest", but I am not sure what that should look like. But having a simple way to persist state would be of great immediate help to me, and to this PR, incidentally. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dstandish edited a comment on issue #6210: [AIRFLOW-5567] BaseReschedulePokeOperator
dstandish edited a comment on issue #6210: [AIRFLOW-5567] BaseReschedulePokeOperator URL: https://github.com/apache/airflow/pull/6210#issuecomment-558030520 > Having a state table will have a fundamental impact on the idempotency of the execution of the tasks. It's optional to use such a thing. Just like it is with XCom. If you don't use it, nothing is changed. > Why would the manual triggering of a dag introduce issues, the execution date will be equal to the moment that it was triggered. I think it should work as well. Because execution_date is run date minus one interval, and `xcom_pull` sorts by execution_date. So, suppose I want to persist state with XCom (which I do in many jobs), and I have a daily job, running at midnight. At end of each run, we push some value to XCom. At start of next job, we retrieve last value and use it somehow. Consider this case: * run 1: 12am D1 * run 2: manually triggered at 8am (exec date is D1 8am; xcom retrieves from run 1) * run 3: 12am D2 * run 4: 12am D3 * run 5: 12am D4 Outcome: * Run 3 will retrive the XCom from run 1, because its execution date is prior to run 2 execution date. * Run 4 retrieves run 2 for same reason. * Run 5 retrieves run 4 (finally things are back in order); run 3 xcom is never retrieved by any job. The schedule interval edge PR would resolve the execution date ordering problem. But if XCom is cleared at start of task, it is remains problematic as a mechanism for state persistence. > Since this will introduce such as a fundamental change to the way operators were intended, being idempotent, I think it would be great to first start an AIP on the topic, so we can have a clear and structured approach. An AIP sounds appropriate. I am just a bit skeptical of the notion that this is some radical change; I would be shocked if stateful processes were not already an extremely common use pattern. Here the goal would be to provide better support for them out of the box. Airflow provides great support for a particular kind of "idempotent" task, but surely it doesn't say this is the only way we can use it! Anyway, I have occasionally rambled on dev list about these issues. I am not sure what the best solution is. I wish there could be clearer and more generalized separation between the concepts of "run date" and "interval of interest", but I am not sure what that should look like. But having a simple way to persist state would be of great immediate help to me, and to this PR, incidentally. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dstandish edited a comment on issue #6210: [AIRFLOW-5567] BaseReschedulePokeOperator
dstandish edited a comment on issue #6210: [AIRFLOW-5567] BaseReschedulePokeOperator URL: https://github.com/apache/airflow/pull/6210#issuecomment-558030520 > Having a state table will have a fundamental impact on the idempotency of the execution of the tasks. It's optional to use such a thing. Just like it is with XCom. If you don't use it, nothing is changed. > Why would the manual triggering of a dag introduce issues, the execution date will be equal to the moment that it was triggered. I think it should work as well. Because execution_date is run date minus one interval. So, suppose I want to persist state with XCom (which I do in many jobs), and I have a daily job, running at midnight. At end of each run, we push some value to XCom. At start of next job, we retrieve last value and use it somehow. Consider this case: * run 1: 12am D1 * run 2: manually triggered at 8am (exec date is D1 8am; xcom retrieves from run 1) * run 3: 12am D2 * run 4: 12am D3 * run 5: 12am D4 Outcome: * Run 3 will retrive the XCom from run 1, because its execution date is prior to run 2 execution date. * Run 4 retrieves run 2 for same reason. * Run 5 retrieves run 4 (finally things are back in order); run 3 xcom is never retrieved by any job. The schedule interval edge PR would resolve the execution date ordering problem. But if XCom is cleared at start of task, it is remains problematic as a mechanism for state persistence. > Since this will introduce such as a fundamental change to the way operators were intended, being idempotent, I think it would be great to first start an AIP on the topic, so we can have a clear and structured approach. An AIP sounds appropriate. I am just a bit skeptical of the notion that this is some radical change; I would be shocked if stateful processes were not already an extremely common use pattern. Here the goal would be to provide better support for them out of the box. Airflow provides great support for a particular kind of "idempotent" task, but surely it doesn't say this is the only way we can use it! Anyway, I have occasionally rambled on dev list about these issues. I am not sure what the best solution is. I wish there could be clearer and more generalized separation between the concepts of "run date" and "interval of interest", but I am not sure what that should look like. But having a simple way to persist state would be of great immediate help to me, and to this PR, incidentally. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dstandish edited a comment on issue #6210: [AIRFLOW-5567] BaseReschedulePokeOperator
dstandish edited a comment on issue #6210: [AIRFLOW-5567] BaseReschedulePokeOperator URL: https://github.com/apache/airflow/pull/6210#issuecomment-558030520 > Having a state table will have a fundamental impact on the idempotency of the execution of the tasks. It's optional to use such a thing. Just like it is with XCom. If you don't use it, nothing is changed. > Why would the manual triggering of a dag introduce issues, the execution date will be equal to the moment that it was triggered. I think it should work as well. Because execution_date is run date minus one interval. So, suppose I want to persist state with XCom (which I do in many jobs), and I have a daily job, running at midnight. At end of each run, we push some value to XCom. At start of next job, we retrieve last value and use it somehow. Consider this case: * run 1: 12am D1 * run 2: manually triggered at 8am (exec date is D1 8am; xcom retrieves from run 1) * run 3: 12am D2 * run 4: 12am D3 * run 5: 12am D4 Outcome: * Run 3 will retrive the XCom from run 1, because its execution date is prior to run 2 execution date. * Run 4 retrieves run 2 for same reason. * Run 5 retrieves run 4 (finally things are back in order); run 3 xcom is never retrieved by any job. The schedule interval edge PR would resolve the execution date ordering problem. But if XCom is cleared at start of task, it is remains problematic as a mechanism for state persistence. > Since this will introduce such as a fundamental change to the way operators were intended, being idempotent, I think it would be great to first start an AIP on the topic, so we can have a clear and structured approach. An AIP sounds reasonable. I am just a bit skeptical of the notion that this is some radical change; I would be shocked if stateful processes were not already an extremely common use pattern. Here the goal would be to provide better support for them out of the box. Airflow provides great support for a particular kind of "idempotent" task, but surely it doesn't say this is the only way we can use it! Anyway, I have occasionally rambled on dev list about these issues. I am not sure what the best solution is. I wish there could be clearer and more generalized separation between the concepts of "run date" and "interval of interest", but I am not sure what that should look like. But having a simple way to persist state would be of great immediate help to me, and to this PR, incidentally. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] dstandish commented on issue #6210: [AIRFLOW-5567] BaseReschedulePokeOperator
dstandish commented on issue #6210: [AIRFLOW-5567] BaseReschedulePokeOperator URL: https://github.com/apache/airflow/pull/6210#issuecomment-558030520 > Having a state table will have a fundamental impact on the idempotency of the execution of the tasks. It's optional to use such a thing. Just like it is with XCom. If you don't use it, nothing is changed. > Why would the manual triggering of a dag introduce issues, the execution date will be equal to the moment that it was triggered. I think it should work as well. Because execution_date is run date minus one interval. So, suppose I want to persist state with XCom (which I do in many jobs), and I have a daily job, running at midnight. At end of each run, we push some value to XCom. At start of next job, we retrieve last value and use it somehow. Consider this case: * run 1: 12am D1 * run 2: manually triggered at 8am (exec date is D1 8am; xcom retrieves from run 1) * run 3: 12am D2 * run 4: 12am D3 * run 5: 12am D3 Outcome: * Run 3 will retrive the XCom from run 1, because its execution date is prior to run 2 execution date. * Run 4 retrieves run 2 for same reason. * Run 5 retrieves run 4 (finally things are back in order); run 3 xcom is never retrieved by any job. The schedule interval edge PR would resolve the execution date ordering problem. But if XCom is cleared at start of task, it is remains problematic as a mechanism for state persistence. > Since this will introduce such as a fundamental change to the way operators were intended, being idempotent, I think it would be great to first start an AIP on the topic, so we can have a clear and structured approach. An AIP sounds reasonable. I am just a bit skeptical of the notion that this is some radical change; I would be shocked if stateful processes were not already an extremely common use pattern. Here the goal would be to provide better support for them out of the box. Airflow provides great support for a particular kind of "idempotent" task, but surely it doesn't say this is the only way we can use it! Anyway, I have occasionally rambled on dev list about these issues. I am not sure what the best solution is. I wish there could be clearer and more generalized separation between the concepts of "run date" and "interval of interest", but I am not sure what that should look like. But having a simple way to persist state would be of great immediate help to me, and to this PR, incidentally. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] OmerJog commented on issue #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run
OmerJog commented on issue #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run URL: https://github.com/apache/airflow/pull/4751#issuecomment-558028258 @amichai07 can you please fix error? ``` == 153) FAIL: test_get_states_count_upstream_ti (tests.ti_deps.deps.test_trigger_rule_dep.TestTriggerRuleDep) -- Traceback (most recent call last): tests/ti_deps/deps/test_trigger_rule_dep.py line 428 in test_get_states_count_upstream_ti (1, 0, 1, 0, 2)) AssertionError: Tuples differ: (1, 0, 1, 1, 2) != (1, 0, 1, 0, 2) First differing element 3: 1 0 - (1, 0, 1, 1, 2) ? ^ + (1, 0, 1, 0, 2) ? ^ ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] Fokko commented on issue #6210: [AIRFLOW-5567] BaseReschedulePokeOperator
Fokko commented on issue #6210: [AIRFLOW-5567] BaseReschedulePokeOperator URL: https://github.com/apache/airflow/pull/6210#issuecomment-558015671 @dstandish Sorry for the late reply. I think having a second table or changing the existing xcom table is just an implementation detail. Having a state table will have a fundamental impact on the idempotency of the execution of the tasks. Why would the manual triggering of a dag introduce issues, the execution date will be equal to the moment that it was triggered. I think it should work as well. Since this will introduce such as a fundamental change to the way operators were intended, being idempotent, I think it would be great to first start an AIP on the topic, so we can have a clear an structured approach. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6596: [AIRFLOW-6004] Untangle Executors class to avoid cyclic imports. Depends on [AIRFLOW-6010]
potiuk commented on issue #6596: [AIRFLOW-6004] Untangle Executors class to avoid cyclic imports. Depends on [AIRFLOW-6010] URL: https://github.com/apache/airflow/pull/6596#issuecomment-558013576 Hey @ashb @kaxil @bolkedebruin @mik-laj -> can you please take a look at this one and see if you can start reviewing it. It fixes the seemingly random "cyclic dependencies" found by pylint. I am fixing the last failing tests, but I think it's worth taking a look even now. I not only fixed the cyclic dependencies - by moving executor-related classes - but I also made the whole `executors` class pylint and mypy compatible. I added all type annotations and happily used variable annotations from python 3.6 which made the executor's code actually ... understandable :). Executors used a lot of tuples to communicate between different objects and I managed to reconstruct the actual types used, I made "aliases" of those types and let mypy + pylint actually verify that they are used properly. Along the way I found that one of the "queues" in execute_async could be removed as it was not used. Pls. take a look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] stale[bot] commented on issue #2460: [AIRFLOW-1424] make the next execution date of DAGs visible
stale[bot] commented on issue #2460: [AIRFLOW-1424] make the next execution date of DAGs visible URL: https://github.com/apache/airflow/pull/2460#issuecomment-558008094 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] msumit commented on issue #6653: [AIRFLOW-6054] Add a command that starts the database consoles
msumit commented on issue #6653: [AIRFLOW-6054] Add a command that starts the database consoles URL: https://github.com/apache/airflow/pull/6653#issuecomment-557995877 nice addition. lgtm. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] XD-DENG commented on a change in pull request #6644: [AIRFLOW-6047] Simplify the logging configuration template
XD-DENG commented on a change in pull request #6644: [AIRFLOW-6047] Simplify
the logging configuration template
URL: https://github.com/apache/airflow/pull/6644#discussion_r349981801
##
File path: airflow/config_templates/airflow_local_settings.py
##
@@ -191,32 +215,6 @@
'json_format': ELASTICSEARCH_JSON_FORMAT,
'json_fields': ELASTICSEARCH_JSON_FIELDS
},
-},
-}
-
-REMOTE_LOGGING = conf.getboolean('core', 'remote_logging')
-
-# Only update the handlers and loggers when CONFIG_PROCESSOR_MANAGER_LOGGER is
set.
-# This is to avoid exceptions when initializing RotatingFileHandler multiple
times
-# in multiple processes.
-if os.environ.get('CONFIG_PROCESSOR_MANAGER_LOGGER') == 'True':
-DEFAULT_LOGGING_CONFIG['handlers'] \
-.update(DEFAULT_DAG_PARSING_LOGGING_CONFIG['handlers'])
-DEFAULT_LOGGING_CONFIG['loggers'] \
-.update(DEFAULT_DAG_PARSING_LOGGING_CONFIG['loggers'])
-
-# Manually create log directory for processor_manager handler as
RotatingFileHandler
-# will only create file but not the directory.
-processor_manager_handler_config =
DEFAULT_DAG_PARSING_LOGGING_CONFIG['handlers'][
-'processor_manager']
-directory = os.path.dirname(processor_manager_handler_config['filename'])
-mkdirs(directory, 0o755)
+}
-if REMOTE_LOGGING and REMOTE_BASE_LOG_FOLDER.startswith('s3://'):
-DEFAULT_LOGGING_CONFIG['handlers'].update(REMOTE_HANDLERS['s3'])
-elif REMOTE_LOGGING and REMOTE_BASE_LOG_FOLDER.startswith('gs://'):
-DEFAULT_LOGGING_CONFIG['handlers'].update(REMOTE_HANDLERS['gcs'])
-elif REMOTE_LOGGING and REMOTE_BASE_LOG_FOLDER.startswith('wasb'):
-DEFAULT_LOGGING_CONFIG['handlers'].update(REMOTE_HANDLERS['wasb'])
-elif REMOTE_LOGGING and ELASTICSEARCH_HOST:
-DEFAULT_LOGGING_CONFIG['handlers'].update(REMOTE_HANDLERS['elasticsearch'])
+DEFAULT_LOGGING_CONFIG['handlers'].update(ELASTIC_REMOTE_HANDLERS)
Review comment:
Do you think it's worthwhile to add a `else:` here? It may help handle cases
like user enters something wrong when they configure (e.g. user makes a typo in
REMOTE_BASE_LOG_FOLDER and it starts with something like upper case "S3:" or
"hs://")
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] XD-DENG commented on a change in pull request #6644: [AIRFLOW-6047] Simplify the logging configuration template
XD-DENG commented on a change in pull request #6644: [AIRFLOW-6047] Simplify
the logging configuration template
URL: https://github.com/apache/airflow/pull/6644#discussion_r349981433
##
File path: airflow/config_templates/airflow_local_settings.py
##
@@ -148,26 +128,61 @@
}
}
-REMOTE_HANDLERS = {
-'s3': {
+# Only update the handlers and loggers when CONFIG_PROCESSOR_MANAGER_LOGGER is
set.
+# This is to avoid exceptions when initializing RotatingFileHandler multiple
times
+# in multiple processes.
+if os.environ.get('CONFIG_PROCESSOR_MANAGER_LOGGER') == 'True':
+DEFAULT_LOGGING_CONFIG['handlers'] \
+.update(DEFAULT_DAG_PARSING_LOGGING_CONFIG['handlers'])
+DEFAULT_LOGGING_CONFIG['loggers'] \
+.update(DEFAULT_DAG_PARSING_LOGGING_CONFIG['loggers'])
+
+# Manually create log directory for processor_manager handler as
RotatingFileHandler
+# will only create file but not the directory.
+processor_manager_handler_config =
DEFAULT_DAG_PARSING_LOGGING_CONFIG['handlers'][
+'processor_manager']
+directory = os.path.dirname(processor_manager_handler_config['filename'])
+mkdirs(directory, 0o755)
+
+# Remote logging configuration
+
+# Storage bucket URL for remote logging
+# S3 buckets should start with "s3://"
+# GCS buckets should start with "gs://"
+# WASB buckets should start with "wasb"
+# just to help Airflow select correct handler
+REMOTE_BASE_LOG_FOLDER = conf.get('core', 'REMOTE_BASE_LOG_FOLDER')
+
+ELASTICSEARCH_HOST = conf.get('elasticsearch', 'HOST')
+
+REMOTE_LOGGING = conf.getboolean('core', 'remote_logging')
+
+if REMOTE_LOGGING and REMOTE_BASE_LOG_FOLDER.startswith('s3://'):
+S3_REMOTE_HANDLERS = {
'task': {
'class': 'airflow.utils.log.s3_task_handler.S3TaskHandler',
'formatter': 'airflow',
Review comment:
`formatter`, `base_log_folder`, and `filename_template` are shared and are
always the same in different handlers.
Wonder if it's a good idea to have a `task template` (inside `formatter`,
`base_log_folder`, and `filename_template` are defined), and create your
`*_REMOTE_HANDLERS` using this template.
This can avoid duplicated lines, and make potential future updating on
`formatter`/`base_log_folder`/`filename_template` easier.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] XD-DENG commented on a change in pull request #6644: [AIRFLOW-6047] Simplify the logging configuration template
XD-DENG commented on a change in pull request #6644: [AIRFLOW-6047] Simplify
the logging configuration template
URL: https://github.com/apache/airflow/pull/6644#discussion_r349981801
##
File path: airflow/config_templates/airflow_local_settings.py
##
@@ -191,32 +215,6 @@
'json_format': ELASTICSEARCH_JSON_FORMAT,
'json_fields': ELASTICSEARCH_JSON_FIELDS
},
-},
-}
-
-REMOTE_LOGGING = conf.getboolean('core', 'remote_logging')
-
-# Only update the handlers and loggers when CONFIG_PROCESSOR_MANAGER_LOGGER is
set.
-# This is to avoid exceptions when initializing RotatingFileHandler multiple
times
-# in multiple processes.
-if os.environ.get('CONFIG_PROCESSOR_MANAGER_LOGGER') == 'True':
-DEFAULT_LOGGING_CONFIG['handlers'] \
-.update(DEFAULT_DAG_PARSING_LOGGING_CONFIG['handlers'])
-DEFAULT_LOGGING_CONFIG['loggers'] \
-.update(DEFAULT_DAG_PARSING_LOGGING_CONFIG['loggers'])
-
-# Manually create log directory for processor_manager handler as
RotatingFileHandler
-# will only create file but not the directory.
-processor_manager_handler_config =
DEFAULT_DAG_PARSING_LOGGING_CONFIG['handlers'][
-'processor_manager']
-directory = os.path.dirname(processor_manager_handler_config['filename'])
-mkdirs(directory, 0o755)
+}
-if REMOTE_LOGGING and REMOTE_BASE_LOG_FOLDER.startswith('s3://'):
-DEFAULT_LOGGING_CONFIG['handlers'].update(REMOTE_HANDLERS['s3'])
-elif REMOTE_LOGGING and REMOTE_BASE_LOG_FOLDER.startswith('gs://'):
-DEFAULT_LOGGING_CONFIG['handlers'].update(REMOTE_HANDLERS['gcs'])
-elif REMOTE_LOGGING and REMOTE_BASE_LOG_FOLDER.startswith('wasb'):
-DEFAULT_LOGGING_CONFIG['handlers'].update(REMOTE_HANDLERS['wasb'])
-elif REMOTE_LOGGING and ELASTICSEARCH_HOST:
-DEFAULT_LOGGING_CONFIG['handlers'].update(REMOTE_HANDLERS['elasticsearch'])
+DEFAULT_LOGGING_CONFIG['handlers'].update(ELASTIC_REMOTE_HANDLERS)
Review comment:
Do you think it's worthwhile to add a `else:` here? It may help handle cases
like user enters something wrong when they configure (e.g. user makes a typo in
REMOTE_BASE_LOG_FOLDER and it starts with something like upper case "S3:" or
"hs://")
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] XD-DENG commented on a change in pull request #6644: [AIRFLOW-6047] Simplify the logging configuration template
XD-DENG commented on a change in pull request #6644: [AIRFLOW-6047] Simplify
the logging configuration template
URL: https://github.com/apache/airflow/pull/6644#discussion_r349981853
##
File path: airflow/config_templates/airflow_local_settings.py
##
@@ -191,32 +215,6 @@
'json_format': ELASTICSEARCH_JSON_FORMAT,
'json_fields': ELASTICSEARCH_JSON_FIELDS
},
-},
-}
-
-REMOTE_LOGGING = conf.getboolean('core', 'remote_logging')
-
-# Only update the handlers and loggers when CONFIG_PROCESSOR_MANAGER_LOGGER is
set.
-# This is to avoid exceptions when initializing RotatingFileHandler multiple
times
-# in multiple processes.
-if os.environ.get('CONFIG_PROCESSOR_MANAGER_LOGGER') == 'True':
-DEFAULT_LOGGING_CONFIG['handlers'] \
-.update(DEFAULT_DAG_PARSING_LOGGING_CONFIG['handlers'])
-DEFAULT_LOGGING_CONFIG['loggers'] \
-.update(DEFAULT_DAG_PARSING_LOGGING_CONFIG['loggers'])
-
-# Manually create log directory for processor_manager handler as
RotatingFileHandler
-# will only create file but not the directory.
-processor_manager_handler_config =
DEFAULT_DAG_PARSING_LOGGING_CONFIG['handlers'][
-'processor_manager']
-directory = os.path.dirname(processor_manager_handler_config['filename'])
-mkdirs(directory, 0o755)
+}
-if REMOTE_LOGGING and REMOTE_BASE_LOG_FOLDER.startswith('s3://'):
-DEFAULT_LOGGING_CONFIG['handlers'].update(REMOTE_HANDLERS['s3'])
-elif REMOTE_LOGGING and REMOTE_BASE_LOG_FOLDER.startswith('gs://'):
-DEFAULT_LOGGING_CONFIG['handlers'].update(REMOTE_HANDLERS['gcs'])
-elif REMOTE_LOGGING and REMOTE_BASE_LOG_FOLDER.startswith('wasb'):
-DEFAULT_LOGGING_CONFIG['handlers'].update(REMOTE_HANDLERS['wasb'])
-elif REMOTE_LOGGING and ELASTICSEARCH_HOST:
-DEFAULT_LOGGING_CONFIG['handlers'].update(REMOTE_HANDLERS['elasticsearch'])
+DEFAULT_LOGGING_CONFIG['handlers'].update(ELASTIC_REMOTE_HANDLERS)
Review comment:
Do you think it's worthwhile to add a `else:` at the end? Inside it we can
handle cases like user enters something wrong when they configure (e.g. user
makes a typo in REMOTE_BASE_LOG_FOLDER and it starts with something like upper
case "S3:" or "hs://")
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] shantanugadgil commented on issue #2708: [AIRFLOW-1746] Add a Nomad operator to trigger job from Airflow
shantanugadgil commented on issue #2708: [AIRFLOW-1746] Add a Nomad operator to trigger job from Airflow URL: https://github.com/apache/airflow/pull/2708#issuecomment-557967329 Should we login/signup on the Apache JIRA and then click "Vote for this issue"? (I have done that as well). Hope this gets implemented soon! :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] shantanugadgil commented on issue #2708: [AIRFLOW-1746] Add a Nomad operator to trigger job from Airflow
shantanugadgil commented on issue #2708: [AIRFLOW-1746] Add a Nomad operator to trigger job from Airflow URL: https://github.com/apache/airflow/pull/2708#issuecomment-557966837 I have added "+1" on this issue, though I see the "stale-bot" marking this as "stale". Is this Github issue and JIRA ticket (https://issues.apache.org/jira/browse/AIRFLOW-5633) still on the radar? :wonder: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] tfindlay-tw commented on a change in pull request #6604: [AIRFLOW-5920] *DRAFT* Neo4j operator and hook
tfindlay-tw commented on a change in pull request #6604: [AIRFLOW-5920] *DRAFT*
Neo4j operator and hook
URL: https://github.com/apache/airflow/pull/6604#discussion_r349969408
##
File path: airflow/contrib/operators/neo4j_operator.py
##
@@ -0,0 +1,92 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.exceptions import AirflowException
+from airflow.contrib.hooks.neo4j_hook import Neo4JHook
+import csv
+
+
+class Neo4JOperator(BaseOperator):
+"""
+Neo4JOperator to interact and perform action on Neo4J graph database.
+This operator is designed to use Neo4J Python driver:
https://neo4j.com/docs/api/python-driver/current/
+
+:param cypher_query: required cypher query to be executed on the Neo4J
database
+:type cypher_query: str
+:param output_filename: required filename to produce with output from the
query
+:type output_filename: str
+:param n4j_conn_id: reference to a pre-defined Neo4J Connection
+:type n4j_conn_id: str
+:param fail_on_no_results: True/False flag to indicate if it should fail
the task if no results
+:type fail_on_no_results: bool
+"""
+_cypher_query = None
+_output_filename = None
+_n4j_conn_id = None
+_fail_on_no_results = None
+
+@apply_defaults
+def __init__(self,
+ cypher_query,
+ output_filename,
+ n4j_conn_id='n4j_default',
+ fail_on_no_results=False,
+ *args,
+ **kwargs):
+super().__init__(*args, **kwargs)
+
+self._output_filename = output_filename
+self._cypher_query = cypher_query
+self._n4j_conn_id = n4j_conn_id
+self._fail_on_no_results = fail_on_no_results
+
+def execute(self, context):
+hook = Neo4JHook(n4j_conn_id=self._n4j_conn_id)
+if self._cypher_query is not None:
+result = hook.run_query(cypher_query=self._cypher_query)
+else:
+raise AirflowException("cypher_query is missing.")
+
+# In some cases, an empty result should fail (where results are
expected)
+if result.peek() is None and self._fail_on_no_results:
+raise AirflowException("Query returned no rows")
+
+row_count = self._make_csv(result)
+
+# Provide some feedback to what was done...
+self.log.info("Saved {0} with {1} rows".format(self._output_filename,
row_count))
+
+# result = 'neo4j.BoltStatementResult' See
https://neo4j.com/docs/api/python-driver/current/results.html
+self.log.info("Processing output with keys: {}".format(result.keys()))
+
+def _make_csv(self, result):
+total_row_count = 0
+
+# Consider available disk space on the Airflow server, maybe support
S3 bucket and bring in the S3 Hook
+with open(self._output_filename, 'w', newline='') as output_file:
+output_writer = csv.DictWriter(output_file,
fieldnames=result.keys())
+output_writer.writeheader()
+
+for total_row_count, row in enumerate(result, start=1):
Review comment:
Only to provide a counter to how many records were processed. It would be
good if there was a more efficient way of doing it, but I dont believe the
BoltStatementResult provides a count for read queries.
Happy to remove if its of a concern, I just felt like feedback can be useful
and this was a way of capturing something.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] tfindlay-tw commented on a change in pull request #6604: [AIRFLOW-5920] *DRAFT* Neo4j operator and hook
tfindlay-tw commented on a change in pull request #6604: [AIRFLOW-5920] *DRAFT* Neo4j operator and hook URL: https://github.com/apache/airflow/pull/6604#discussion_r349969435 ## File path: docs/howto/operator/neo4j.rst ## @@ -0,0 +1,72 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +Neo4j Operator +== + +This operator enables Airflow DAGs to execute cypher queries against a Neo4j (or ONgDB) graph database. + +The results of the query execution will be written to a CSV file on disk. Please consider available space +on the Airflow worker if your query can return a large result. + +This operator can be used in conjunction with the S3 operator or email operator to process the results. + +See the :ref:`Operators Concepts ` documentation and the :doc:`Operators API Reference <../../_api/index>` for more information. + +Prerequisite Tasks +^^ +To use this operator you must define a connection to your Neo4J/ONgDB database via: + + *Admin* -> *Connections* -> *Create* + +The connection must set the following properties: + +- login +- password +- host +- port + +The connection name is then used in the DAG to reference this definition. + +Basic Usage +^^^ +Use the :class:`~airflow/contrib/operators/neo4j_operator.Neo4JOperator` to execute cyhpher query: + +.. exampleinclude:: ../../../airflow/contrib/example_dags/example_neo4j_operator.py +:language: python +:dedent: 4 +:start-after: [START howto_operator_dingding] +:end-before: [END howto_operator_dingding] Review comment: Opps, this should be fixed now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] tfindlay-tw commented on a change in pull request #6604: [AIRFLOW-5920] *DRAFT* Neo4j operator and hook
tfindlay-tw commented on a change in pull request #6604: [AIRFLOW-5920] *DRAFT*
Neo4j operator and hook
URL: https://github.com/apache/airflow/pull/6604#discussion_r349969134
##
File path: airflow/contrib/operators/neo4j_operator.py
##
@@ -0,0 +1,92 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.exceptions import AirflowException
+from airflow.contrib.hooks.neo4j_hook import Neo4JHook
+import csv
+
+
+class Neo4JOperator(BaseOperator):
+"""
+Neo4JOperator to interact and perform action on Neo4J graph database.
+This operator is designed to use Neo4J Python driver:
https://neo4j.com/docs/api/python-driver/current/
+
+:param cypher_query: required cypher query to be executed on the Neo4J
database
+:type cypher_query: str
+:param output_filename: required filename to produce with output from the
query
+:type output_filename: str
+:param n4j_conn_id: reference to a pre-defined Neo4J Connection
+:type n4j_conn_id: str
+:param fail_on_no_results: True/False flag to indicate if it should fail
the task if no results
+:type fail_on_no_results: bool
+"""
+_cypher_query = None
+_output_filename = None
+_n4j_conn_id = None
+_fail_on_no_results = None
+
+@apply_defaults
+def __init__(self,
+ cypher_query,
+ output_filename,
+ n4j_conn_id='n4j_default',
+ fail_on_no_results=False,
+ *args,
+ **kwargs):
+super().__init__(*args, **kwargs)
+
+self._output_filename = output_filename
+self._cypher_query = cypher_query
+self._n4j_conn_id = n4j_conn_id
+self._fail_on_no_results = fail_on_no_results
+
+def execute(self, context):
+hook = Neo4JHook(n4j_conn_id=self._n4j_conn_id)
+if self._cypher_query is not None:
+result = hook.run_query(cypher_query=self._cypher_query)
+else:
+raise AirflowException("cypher_query is missing.")
Review comment:
Sure, at one point, I considered making it optional. Queries could be read
in from files or other sources, but then simplified it to just accept a string.
Have removed this to reflect how it is designed to operate right now.
Complexity can be added later if needed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] tfindlay-tw commented on a change in pull request #6604: [AIRFLOW-5920] *DRAFT* Neo4j operator and hook
tfindlay-tw commented on a change in pull request #6604: [AIRFLOW-5920] *DRAFT*
Neo4j operator and hook
URL: https://github.com/apache/airflow/pull/6604#discussion_r349968925
##
File path: airflow/contrib/hooks/neo4j_hook.py
##
@@ -0,0 +1,93 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+from neo4j import GraphDatabase
+
+
+class Neo4JHook(BaseHook):
+"""
+Interact with Neo4J.
+This class is a thin wrapper around the neo4j python library.
+"""
+_n4j_conn_id = None
+
+def __init__(self, n4j_conn_id='n4j_default', *args, **kwargs):
+super().__init__()
+self._n4j_conn_id = n4j_conn_id
+
+@staticmethod
+def get_config(n4j_conn_id):
+"""
+Obtain the Username + Password from the Airflow connection definition
+Store them in _config dictionary as:
+ credentials = a tuple of username/password eg. ("username",
"password")
+ host = String for Neo4J URI eg. "bolt://1.1.1.1:7687"
+:return: dictionary with configuration values
+"""
+config = {}
+if n4j_conn_id:
+# Initialize with empty dictionary
+connection_object = Neo4JHook.get_connection(n4j_conn_id)
+if connection_object.login and connection_object.host:
+config['credentials'] = connection_object.login,
connection_object.password
+config['host'] =
"bolt://{0}:{1}".format(connection_object.host, connection_object.port)
+else:
+raise AirflowException("No Neo4J connection:
{}".format(n4j_conn_id))
+
+return config
+
+@staticmethod
+def get_driver(config):
+"""
+Establish a TCP connection to the server
+"""
+# Check if we already have a driver we can re-use before creating a
new one
+return GraphDatabase.driver(
+uri=config['host'],
+auth=config['credentials']
+)
+
+@staticmethod
+def get_session(driver):
+"""
+Get a neo4j.session from the driver.
+"""
+# Check if we already have a session we can re-use before creating a
new one
+return driver.session()
+
+def run_query(self, cypher_query, parameters=None):
+"""
+Uses a session to execute submit a query for execution
+:param cypher_query: Cypher query eg. MATCH (a) RETURN (a)
+:param parameters: Optional list of parameters to use with the query
+:return: neo4j.BoltStatementResult see
https://neo4j.com/docs/api/python-driver/current/results.html
+"""
+a = Neo4JHook.get_config(self._n4j_conn_id)
+b = Neo4JHook.get_driver(a)
+c = Neo4JHook.get_session(b)
Review comment:
Yes, apologies, I did name them better initially, but during testing I did
this, they should be fixed up now.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6656: [AIRFLOW-6057] Update template_fields of the PythonSensor
codecov-io edited a comment on issue #6656: [AIRFLOW-6057] Update template_fields of the PythonSensor URL: https://github.com/apache/airflow/pull/6656#issuecomment-557940810 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6656?src=pr=h1) Report > Merging [#6656](https://codecov.io/gh/apache/airflow/pull/6656?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/db4be19b80a65913edb5ab634226cf60ea8bffea?src=pr=desc) will **decrease** coverage by `0.33%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6656?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6656 +/- ## == - Coverage 83.82% 83.48% -0.34% == Files 672 672 Lines 3756037560 == - Hits3148331357 -126 - Misses 6077 6203 +126 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6656?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/sensors/python\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6656/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvcHl0aG9uX3NlbnNvci5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6656/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6656/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6656/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6656/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6656/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6656/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `85% <0%> (-5%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6656/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6656/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.15% <0%> (-0.33%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6656?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6656?src=pr=footer). Last update [db4be19...e5cc54a](https://codecov.io/gh/apache/airflow/pull/6656?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6656: [AIRFLOW-6057] Update template_fields of the PythonSensor
codecov-io commented on issue #6656: [AIRFLOW-6057] Update template_fields of the PythonSensor URL: https://github.com/apache/airflow/pull/6656#issuecomment-557940810 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6656?src=pr=h1) Report > Merging [#6656](https://codecov.io/gh/apache/airflow/pull/6656?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/db4be19b80a65913edb5ab634226cf60ea8bffea?src=pr=desc) will **decrease** coverage by `0.33%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6656?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6656 +/- ## == - Coverage 83.82% 83.48% -0.34% == Files 672 672 Lines 3756037560 == - Hits3148331357 -126 - Misses 6077 6203 +126 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6656?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/sensors/python\_sensor.py](https://codecov.io/gh/apache/airflow/pull/6656/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL3NlbnNvcnMvcHl0aG9uX3NlbnNvci5weQ==) | `100% <100%> (ø)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6656/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6656/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6656/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6656/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6656/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6656/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `85% <0%> (-5%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6656/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6656/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.15% <0%> (-0.33%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6656?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6656?src=pr=footer). Last update [db4be19...e5cc54a](https://codecov.io/gh/apache/airflow/pull/6656?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6057) Add op_args and op_kwargs in PythonSensor templated fields
[
https://issues.apache.org/jira/browse/AIRFLOW-6057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981198#comment-16981198
]
ASF GitHub Bot commented on AIRFLOW-6057:
-
mustafagok commented on pull request #6656: [AIRFLOW-6057] Update
template_fields of the PythonSensor
URL: https://github.com/apache/airflow/pull/6656
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-6057
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Allow passing Jinja templated arguments to the python_callable of the
PythonSensor (just like in PythonOperator) by adding op_args and op_kwargs to
the template_fields of the PythonSensor.
Also fix the test_python_callable_keyword_arguments_are_templatized test of
the PythonOperator.
### Tests
- [x] My PR adds the following unit tests:
- test_python_callable_arguments_are_templatized for op_args,
- test_python_callable_keyword_arguments_are_templatized for op_kwargs
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add op_args and op_kwargs in PythonSensor templated fields
> --
>
> Key: AIRFLOW-6057
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6057
> Project: Apache Airflow
> Issue Type: Improvement
> Components: contrib, operators
>Affects Versions: 1.10.7
>Reporter: Mustafa Gök
>Assignee: Mustafa Gök
>Priority: Minor
>
> Similar Issue: AIRFLOW-1814
> Let's say we have a method that returns True or False and we want to use it
> with PythonSensor, but since the method is used in other applications, we
> don't want to edit this method by adding some XCom pulling code.
> If op_args and op_kwargs are added to the template_fields of the
> PythonSensor, Jinja template can be used.
>
> Note: I am not sure about Components and Affects Versions fields of this
> issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] mustafagok opened a new pull request #6656: [AIRFLOW-6057] Update template_fields of the PythonSensor
mustafagok opened a new pull request #6656: [AIRFLOW-6057] Update
template_fields of the PythonSensor
URL: https://github.com/apache/airflow/pull/6656
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-6057
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Allow passing Jinja templated arguments to the python_callable of the
PythonSensor (just like in PythonOperator) by adding op_args and op_kwargs to
the template_fields of the PythonSensor.
Also fix the test_python_callable_keyword_arguments_are_templatized test of
the PythonOperator.
### Tests
- [x] My PR adds the following unit tests:
- test_python_callable_arguments_are_templatized for op_args,
- test_python_callable_keyword_arguments_are_templatized for op_kwargs
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] dstandish commented on issue #6210: [AIRFLOW-5567] BaseReschedulePokeOperator
dstandish commented on issue #6210: [AIRFLOW-5567] BaseReschedulePokeOperator URL: https://github.com/apache/airflow/pull/6210#issuecomment-557931240 So there were concerns with @Fokko's xcom change re idempotency. I think it makes sense to create second table, very similar to xcom, but designed specifically to support stateful tasks. The table could perhaps be called TaskState. This task state should not be pegged to a specific execution date, because execution date only really makes sense for non-stateful tasks. And execution date can be out of sequence with actual run time. I think it might make sense to make it so we don't do updates: when state changes, we insert a new record with the current state. Primary key would be dag id / task id / timestamp. To get current state, we get the last record for the dag / task. It's possible we could allow state to be namespaced under task id with a column `key` like is done with XCom but I don't think it's necessary. I previously shared the concern, why create another table that is almost identical to xcom. But the reality is XCom is problematic for stateful tasks in a number of ways. Obviously there is the clearing / idempotency issue. But additionally if you use trigger dag, with XCom your next scheduled run won't get current state because it sorts by execution_date. WDYT? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6057) Add op_args and op_kwargs in PythonSensor templated fields
Mustafa Gök created AIRFLOW-6057: Summary: Add op_args and op_kwargs in PythonSensor templated fields Key: AIRFLOW-6057 URL: https://issues.apache.org/jira/browse/AIRFLOW-6057 Project: Apache Airflow Issue Type: Improvement Components: contrib, operators Affects Versions: 1.10.7 Reporter: Mustafa Gök Assignee: Mustafa Gök Similar Issue: AIRFLOW-1814 Let's say we have a method that returns True or False and we want to use it with PythonSensor, but since the method is used in other applications, we don't want to edit this method by adding some XCom pulling code. If op_args and op_kwargs are added to the template_fields of the PythonSensor, Jinja template can be used. Note: I am not sure about Components and Affects Versions fields of this issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6604: [AIRFLOW-5920] *DRAFT* Neo4j operator and hook
nuclearpinguin commented on a change in pull request #6604: [AIRFLOW-5920]
*DRAFT* Neo4j operator and hook
URL: https://github.com/apache/airflow/pull/6604#discussion_r349943669
##
File path: airflow/contrib/operators/neo4j_operator.py
##
@@ -0,0 +1,92 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.exceptions import AirflowException
+from airflow.contrib.hooks.neo4j_hook import Neo4JHook
+import csv
+
+
+class Neo4JOperator(BaseOperator):
+"""
+Neo4JOperator to interact and perform action on Neo4J graph database.
+This operator is designed to use Neo4J Python driver:
https://neo4j.com/docs/api/python-driver/current/
+
+:param cypher_query: required cypher query to be executed on the Neo4J
database
+:type cypher_query: str
+:param output_filename: required filename to produce with output from the
query
+:type output_filename: str
+:param n4j_conn_id: reference to a pre-defined Neo4J Connection
+:type n4j_conn_id: str
+:param fail_on_no_results: True/False flag to indicate if it should fail
the task if no results
+:type fail_on_no_results: bool
+"""
+_cypher_query = None
+_output_filename = None
+_n4j_conn_id = None
+_fail_on_no_results = None
+
+@apply_defaults
+def __init__(self,
+ cypher_query,
+ output_filename,
+ n4j_conn_id='n4j_default',
+ fail_on_no_results=False,
+ *args,
+ **kwargs):
+super().__init__(*args, **kwargs)
+
+self._output_filename = output_filename
+self._cypher_query = cypher_query
+self._n4j_conn_id = n4j_conn_id
+self._fail_on_no_results = fail_on_no_results
+
+def execute(self, context):
+hook = Neo4JHook(n4j_conn_id=self._n4j_conn_id)
+if self._cypher_query is not None:
+result = hook.run_query(cypher_query=self._cypher_query)
+else:
+raise AirflowException("cypher_query is missing.")
Review comment:
This try / except is unnecessary because `cypher_query` is required in
constructor. It's also good idea to skip `_`in attributes. I would also suggest
adding `template_fields` to the operator.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6604: [AIRFLOW-5920] *DRAFT* Neo4j operator and hook
nuclearpinguin commented on a change in pull request #6604: [AIRFLOW-5920]
*DRAFT* Neo4j operator and hook
URL: https://github.com/apache/airflow/pull/6604#discussion_r349943764
##
File path: airflow/contrib/operators/neo4j_operator.py
##
@@ -0,0 +1,92 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.exceptions import AirflowException
+from airflow.contrib.hooks.neo4j_hook import Neo4JHook
+import csv
+
+
+class Neo4JOperator(BaseOperator):
+"""
+Neo4JOperator to interact and perform action on Neo4J graph database.
+This operator is designed to use Neo4J Python driver:
https://neo4j.com/docs/api/python-driver/current/
+
+:param cypher_query: required cypher query to be executed on the Neo4J
database
+:type cypher_query: str
+:param output_filename: required filename to produce with output from the
query
+:type output_filename: str
+:param n4j_conn_id: reference to a pre-defined Neo4J Connection
+:type n4j_conn_id: str
+:param fail_on_no_results: True/False flag to indicate if it should fail
the task if no results
+:type fail_on_no_results: bool
+"""
+_cypher_query = None
+_output_filename = None
+_n4j_conn_id = None
+_fail_on_no_results = None
+
+@apply_defaults
+def __init__(self,
+ cypher_query,
+ output_filename,
+ n4j_conn_id='n4j_default',
+ fail_on_no_results=False,
+ *args,
+ **kwargs):
+super().__init__(*args, **kwargs)
+
+self._output_filename = output_filename
+self._cypher_query = cypher_query
+self._n4j_conn_id = n4j_conn_id
+self._fail_on_no_results = fail_on_no_results
+
+def execute(self, context):
+hook = Neo4JHook(n4j_conn_id=self._n4j_conn_id)
+if self._cypher_query is not None:
+result = hook.run_query(cypher_query=self._cypher_query)
+else:
+raise AirflowException("cypher_query is missing.")
+
+# In some cases, an empty result should fail (where results are
expected)
+if result.peek() is None and self._fail_on_no_results:
+raise AirflowException("Query returned no rows")
+
+row_count = self._make_csv(result)
+
+# Provide some feedback to what was done...
+self.log.info("Saved {0} with {1} rows".format(self._output_filename,
row_count))
+
+# result = 'neo4j.BoltStatementResult' See
https://neo4j.com/docs/api/python-driver/current/results.html
+self.log.info("Processing output with keys: {}".format(result.keys()))
+
+def _make_csv(self, result):
+total_row_count = 0
+
+# Consider available disk space on the Airflow server, maybe support
S3 bucket and bring in the S3 Hook
+with open(self._output_filename, 'w', newline='') as output_file:
+output_writer = csv.DictWriter(output_file,
fieldnames=result.keys())
+output_writer.writeheader()
+
+for total_row_count, row in enumerate(result, start=1):
Review comment:
Is there any reason for the enumerate?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6604: [AIRFLOW-5920] *DRAFT* Neo4j operator and hook
nuclearpinguin commented on a change in pull request #6604: [AIRFLOW-5920] *DRAFT* Neo4j operator and hook URL: https://github.com/apache/airflow/pull/6604#discussion_r349943840 ## File path: docs/howto/operator/neo4j.rst ## @@ -0,0 +1,72 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. + +Neo4j Operator +== + +This operator enables Airflow DAGs to execute cypher queries against a Neo4j (or ONgDB) graph database. + +The results of the query execution will be written to a CSV file on disk. Please consider available space +on the Airflow worker if your query can return a large result. + +This operator can be used in conjunction with the S3 operator or email operator to process the results. + +See the :ref:`Operators Concepts ` documentation and the :doc:`Operators API Reference <../../_api/index>` for more information. + +Prerequisite Tasks +^^ +To use this operator you must define a connection to your Neo4J/ONgDB database via: + + *Admin* -> *Connections* -> *Create* + +The connection must set the following properties: + +- login +- password +- host +- port + +The connection name is then used in the DAG to reference this definition. + +Basic Usage +^^^ +Use the :class:`~airflow/contrib/operators/neo4j_operator.Neo4JOperator` to execute cyhpher query: + +.. exampleinclude:: ../../../airflow/contrib/example_dags/example_neo4j_operator.py +:language: python +:dedent: 4 +:start-after: [START howto_operator_dingding] +:end-before: [END howto_operator_dingding] Review comment: Those start/end will need to be adjusted to build docs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] nuclearpinguin commented on a change in pull request #6604: [AIRFLOW-5920] *DRAFT* Neo4j operator and hook
nuclearpinguin commented on a change in pull request #6604: [AIRFLOW-5920]
*DRAFT* Neo4j operator and hook
URL: https://github.com/apache/airflow/pull/6604#discussion_r349943509
##
File path: airflow/contrib/hooks/neo4j_hook.py
##
@@ -0,0 +1,93 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook import BaseHook
+from neo4j import GraphDatabase
+
+
+class Neo4JHook(BaseHook):
+"""
+Interact with Neo4J.
+This class is a thin wrapper around the neo4j python library.
+"""
+_n4j_conn_id = None
+
+def __init__(self, n4j_conn_id='n4j_default', *args, **kwargs):
+super().__init__()
+self._n4j_conn_id = n4j_conn_id
+
+@staticmethod
+def get_config(n4j_conn_id):
+"""
+Obtain the Username + Password from the Airflow connection definition
+Store them in _config dictionary as:
+ credentials = a tuple of username/password eg. ("username",
"password")
+ host = String for Neo4J URI eg. "bolt://1.1.1.1:7687"
+:return: dictionary with configuration values
+"""
+config = {}
+if n4j_conn_id:
+# Initialize with empty dictionary
+connection_object = Neo4JHook.get_connection(n4j_conn_id)
+if connection_object.login and connection_object.host:
+config['credentials'] = connection_object.login,
connection_object.password
+config['host'] =
"bolt://{0}:{1}".format(connection_object.host, connection_object.port)
+else:
+raise AirflowException("No Neo4J connection:
{}".format(n4j_conn_id))
+
+return config
+
+@staticmethod
+def get_driver(config):
+"""
+Establish a TCP connection to the server
+"""
+# Check if we already have a driver we can re-use before creating a
new one
+return GraphDatabase.driver(
+uri=config['host'],
+auth=config['credentials']
+)
+
+@staticmethod
+def get_session(driver):
+"""
+Get a neo4j.session from the driver.
+"""
+# Check if we already have a session we can re-use before creating a
new one
+return driver.session()
+
+def run_query(self, cypher_query, parameters=None):
+"""
+Uses a session to execute submit a query for execution
+:param cypher_query: Cypher query eg. MATCH (a) RETURN (a)
+:param parameters: Optional list of parameters to use with the query
+:return: neo4j.BoltStatementResult see
https://neo4j.com/docs/api/python-driver/current/results.html
+"""
+a = Neo4JHook.get_config(self._n4j_conn_id)
+b = Neo4JHook.get_driver(a)
+c = Neo4JHook.get_session(b)
Review comment:
Please use more meaningful variables' names.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6651: [WIP][AIRFLOW-XXXX] Add info endpoint to experimental api
codecov-io edited a comment on issue #6651: [WIP][AIRFLOW-] Add info endpoint to experimental api URL: https://github.com/apache/airflow/pull/6651#issuecomment-557893722 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=h1) Report > Merging [#6651](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/db4be19b80a65913edb5ab634226cf60ea8bffea?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6651 +/- ## == + Coverage 83.82% 83.82% +<.01% == Files 672 672 Lines 3756037564 +4 == + Hits3148331487 +4 Misses 6077 6077 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/api/experimental/endpoints.py](https://codecov.io/gh/apache/airflow/pull/6651/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvYXBpL2V4cGVyaW1lbnRhbC9lbmRwb2ludHMucHk=) | `88.71% <100%> (+0.23%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=footer). Last update [db4be19...c7d8811](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6651: [WIP][AIRFLOW-XXXX] Add info endpoint to experimental api
codecov-io edited a comment on issue #6651: [WIP][AIRFLOW-] Add info endpoint to experimental api URL: https://github.com/apache/airflow/pull/6651#issuecomment-557893722 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=h1) Report > Merging [#6651](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/db4be19b80a65913edb5ab634226cf60ea8bffea?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6651 +/- ## == + Coverage 83.82% 83.82% +<.01% == Files 672 672 Lines 3756037564 +4 == + Hits3148331487 +4 Misses 6077 6077 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/api/experimental/endpoints.py](https://codecov.io/gh/apache/airflow/pull/6651/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvYXBpL2V4cGVyaW1lbnRhbC9lbmRwb2ludHMucHk=) | `88.71% <100%> (+0.23%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=footer). Last update [db4be19...c7d8811](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6654: AIRFLOW-6055: Option for exponential backoff in Sensors
codecov-io commented on issue #6654: AIRFLOW-6055: Option for exponential backoff in Sensors URL: https://github.com/apache/airflow/pull/6654#issuecomment-557917127 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6654?src=pr=h1) Report > Merging [#6654](https://codecov.io/gh/apache/airflow/pull/6654?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/db4be19b80a65913edb5ab634226cf60ea8bffea?src=pr=desc) will **decrease** coverage by `0.32%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6654?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6654 +/- ## = - Coverage 83.82% 83.5% -0.33% = Files 672 672 Lines 37560 37576 +16 = - Hits31483 31376 -107 - Misses 60776200 +123 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6654?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/sensors/base\_sensor\_operator.py](https://codecov.io/gh/apache/airflow/pull/6654/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL2Jhc2Vfc2Vuc29yX29wZXJhdG9yLnB5) | `98.57% <100%> (+0.42%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6654/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6654/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6654/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6654/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6654/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6654/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6654/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `57.99% <0%> (-0.5%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6654?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6654?src=pr=footer). Last update [db4be19...a1eab04](https://codecov.io/gh/apache/airflow/pull/6654?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6654: AIRFLOW-6055: Option for exponential backoff in Sensors
codecov-io edited a comment on issue #6654: AIRFLOW-6055: Option for exponential backoff in Sensors URL: https://github.com/apache/airflow/pull/6654#issuecomment-557917127 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6654?src=pr=h1) Report > Merging [#6654](https://codecov.io/gh/apache/airflow/pull/6654?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/db4be19b80a65913edb5ab634226cf60ea8bffea?src=pr=desc) will **decrease** coverage by `0.32%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6654?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6654 +/- ## = - Coverage 83.82% 83.5% -0.33% = Files 672 672 Lines 37560 37576 +16 = - Hits31483 31376 -107 - Misses 60776200 +123 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6654?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/sensors/base\_sensor\_operator.py](https://codecov.io/gh/apache/airflow/pull/6654/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL2Jhc2Vfc2Vuc29yX29wZXJhdG9yLnB5) | `98.57% <100%> (+0.42%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6654/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6654/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6654/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6654/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6654/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6654/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6654/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `57.99% <0%> (-0.5%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6654?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6654?src=pr=footer). Last update [db4be19...a1eab04](https://codecov.io/gh/apache/airflow/pull/6654?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] stale[bot] commented on issue #2708: [AIRFLOW-1746] Add a Nomad operator to trigger job from Airflow
stale[bot] commented on issue #2708: [AIRFLOW-1746] Add a Nomad operator to trigger job from Airflow URL: https://github.com/apache/airflow/pull/2708#issuecomment-557916122 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] aviemzur commented on issue #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id
aviemzur commented on issue #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id URL: https://github.com/apache/airflow/pull/6655#issuecomment-557908143 R: @kaxil This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id
codecov-io edited a comment on issue #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id URL: https://github.com/apache/airflow/pull/6655#issuecomment-557906816 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6655?src=pr=h1) Report > Merging [#6655](https://codecov.io/gh/apache/airflow/pull/6655?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/db4be19b80a65913edb5ab634226cf60ea8bffea?src=pr=desc) will **decrease** coverage by `0.31%`. > The diff coverage is `94.11%`. [](https://codecov.io/gh/apache/airflow/pull/6655?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6655 +/- ## = - Coverage 83.82% 83.5% -0.32% = Files 672 672 Lines 37560 37575 +15 = - Hits31483 31377 -106 - Misses 60776198 +121 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6655?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/hooks/emr\_hook.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2Vtcl9ob29rLnB5) | `96.29% <100%> (+1.85%)` | :arrow_up: | | [...irflow/contrib/operators/emr\_add\_steps\_operator.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9lbXJfYWRkX3N0ZXBzX29wZXJhdG9yLnB5) | `93.1% <87.5%> (-2.55%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6655?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6655?src=pr=footer). Last update [db4be19...c946dce](https://codecov.io/gh/apache/airflow/pull/6655?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id
codecov-io edited a comment on issue #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id URL: https://github.com/apache/airflow/pull/6655#issuecomment-557906816 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6655?src=pr=h1) Report > Merging [#6655](https://codecov.io/gh/apache/airflow/pull/6655?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/db4be19b80a65913edb5ab634226cf60ea8bffea?src=pr=desc) will **decrease** coverage by `0.31%`. > The diff coverage is `94.11%`. [](https://codecov.io/gh/apache/airflow/pull/6655?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6655 +/- ## = - Coverage 83.82% 83.5% -0.32% = Files 672 672 Lines 37560 37575 +15 = - Hits31483 31377 -106 - Misses 60776198 +121 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6655?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/hooks/emr\_hook.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2Vtcl9ob29rLnB5) | `96.29% <100%> (+1.85%)` | :arrow_up: | | [...irflow/contrib/operators/emr\_add\_steps\_operator.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9lbXJfYWRkX3N0ZXBzX29wZXJhdG9yLnB5) | `93.1% <87.5%> (-2.55%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6655?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6655?src=pr=footer). Last update [db4be19...c946dce](https://codecov.io/gh/apache/airflow/pull/6655?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id
codecov-io commented on issue #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id URL: https://github.com/apache/airflow/pull/6655#issuecomment-557906816 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6655?src=pr=h1) Report > Merging [#6655](https://codecov.io/gh/apache/airflow/pull/6655?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/db4be19b80a65913edb5ab634226cf60ea8bffea?src=pr=desc) will **decrease** coverage by `0.31%`. > The diff coverage is `94.11%`. [](https://codecov.io/gh/apache/airflow/pull/6655?src=pr=tree) ```diff @@Coverage Diff@@ ## master #6655 +/- ## = - Coverage 83.82% 83.5% -0.32% = Files 672 672 Lines 37560 37575 +15 = - Hits31483 31377 -106 - Misses 60776198 +121 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6655?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/contrib/hooks/emr\_hook.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL2hvb2tzL2Vtcl9ob29rLnB5) | `96.29% <100%> (+1.85%)` | :arrow_up: | | [...irflow/contrib/operators/emr\_add\_steps\_operator.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9lbXJfYWRkX3N0ZXBzX29wZXJhdG9yLnB5) | `93.1% <87.5%> (-2.55%)` | :arrow_down: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6655/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `89.13% <0%> (-3.63%)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6655?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6655?src=pr=footer). Last update [db4be19...c946dce](https://codecov.io/gh/apache/airflow/pull/6655?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6056) Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id
[
https://issues.apache.org/jira/browse/AIRFLOW-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981120#comment-16981120
]
ASF GitHub Bot commented on AIRFLOW-6056:
-
aviemzur commented on pull request #6655: [AIRFLOW-6056] Allow
EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id
URL: https://github.com/apache/airflow/pull/6655
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Allow EmrAddStepsOperator to accept job_flow_name as alternative to
job_flow_id
This is useful for when you have a persistent EMR cluster for testing or for
when you have repeated scheduled jobs with a small interval between them.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Allow EmrAddStepsOperator to accept job_flow_name as alternative to
> job_flow_id
> ---
>
> Key: AIRFLOW-6056
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6056
> Project: Apache Airflow
> Issue Type: New Feature
> Components: contrib
>Affects Versions: 1.10.6
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>Priority: Major
>
> Allow EmrAddStepsOperator to accept job_flow_name as alternative to
> job_flow_id
> This is useful for when you have a persistent EMR cluster for testing or for
> when you have repeated scheduled jobs with a small interval between them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6056) Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id
[ https://issues.apache.org/jira/browse/AIRFLOW-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981119#comment-16981119 ] ASF GitHub Bot commented on AIRFLOW-6056: - aviemzur commented on pull request #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id URL: https://github.com/apache/airflow/pull/6655 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Allow EmrAddStepsOperator to accept job_flow_name as alternative to > job_flow_id > --- > > Key: AIRFLOW-6056 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6056 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Affects Versions: 1.10.6 >Reporter: Aviem Zur >Assignee: Aviem Zur >Priority: Major > > Allow EmrAddStepsOperator to accept job_flow_name as alternative to > job_flow_id > This is useful for when you have a persistent EMR cluster for testing or for > when you have repeated scheduled jobs with a small interval between them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] aviemzur closed pull request #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id
aviemzur closed pull request #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id URL: https://github.com/apache/airflow/pull/6655 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] aviemzur opened a new pull request #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id
aviemzur opened a new pull request #6655: [AIRFLOW-6056] Allow
EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id
URL: https://github.com/apache/airflow/pull/6655
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [x] Here are some details about my PR, including screenshots of any UI
changes:
Allow EmrAddStepsOperator to accept job_flow_name as alternative to
job_flow_id
This is useful for when you have a persistent EMR cluster for testing or for
when you have repeated scheduled jobs with a small interval between them.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #6564: [AIRFLOW-5911] Simplify lineage API and improve robustness
potiuk commented on issue #6564: [AIRFLOW-5911] Simplify lineage API and improve robustness URL: https://github.com/apache/airflow/pull/6564#issuecomment-557900655 FYI @bolkedebruin -> I am working on permanent fix to the seemingly randomly appearing cyclic imports. They are real cycles of dependencies we have, but they are somewhat hidden until you have pylint run on all code (and the more pylint we enable the more of those cycles we will find I think). I hope to heave a fix that gets rid of all those (known so far) cyclic dependencies that are hiding in our code. First step (remove cycles from serialization) is here: #6601 (already passed Travis) and second (remove cycles from executors) here (#6596) - coming soon @ashb @kaxil -> this is exactly the case I was explaining, the cycles are appearing randomly in some travis builds because of sequence/subsets of files passed to pylint processes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] amichai07 commented on a change in pull request #4751: [AIRFLOW-3607] collected trigger rule dep check per dag run
amichai07 commented on a change in pull request #4751: [AIRFLOW-3607] collected
trigger rule dep check per dag run
URL: https://github.com/apache/airflow/pull/4751#discussion_r349927589
##
File path: airflow/jobs/scheduler_job.py
##
@@ -721,25 +721,14 @@ def _process_task_instances(self, dag,
task_instances_list, session=None):
if run.state == State.RUNNING:
make_transient(run)
active_dag_runs.append(run)
-
-for run in active_dag_runs:
-self.log.debug("Examining active DAG run: %s", run)
-tis = run.get_task_instances(state=SCHEDULEABLE_STATES)
-
-# this loop is quite slow as it uses are_dependencies_met for
-# every task (in ti.is_runnable). This is also called in
-# update_state above which has already checked these tasks
-for ti in tis:
-task = dag.get_task(ti.task_id)
-
-# fixme: ti.task is transient but needs to be set
-ti.task = task
-
-if ti.are_dependencies_met(
-dep_context=DepContext(flag_upstream_failed=True),
-session=session):
+self.log.debug("Examining active DAG run: %s", run)
+for ti in run.ready_tis:
Review comment:
yes :)
However just few tests use it for the state and it is kind of an easy change
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6651: [WIP][AIRFLOW-XXXX] Add info endpoint to experimental api
codecov-io edited a comment on issue #6651: [WIP][AIRFLOW-] Add info endpoint to experimental api URL: https://github.com/apache/airflow/pull/6651#issuecomment-557893722 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=h1) Report > Merging [#6651](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/db4be19b80a65913edb5ab634226cf60ea8bffea?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6651 +/- ## == + Coverage 83.82% 83.82% +<.01% == Files 672 672 Lines 3756037564 +4 == + Hits3148331487 +4 Misses 6077 6077 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/api/experimental/endpoints.py](https://codecov.io/gh/apache/airflow/pull/6651/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvYXBpL2V4cGVyaW1lbnRhbC9lbmRwb2ludHMucHk=) | `88.71% <100%> (+0.23%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=footer). Last update [db4be19...ce2e499](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6651: [WIP][AIRFLOW-XXXX] Add info endpoint to experimental api
codecov-io edited a comment on issue #6651: [WIP][AIRFLOW-] Add info endpoint to experimental api URL: https://github.com/apache/airflow/pull/6651#issuecomment-557893722 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=h1) Report > Merging [#6651](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/db4be19b80a65913edb5ab634226cf60ea8bffea?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6651 +/- ## == + Coverage 83.82% 83.82% +<.01% == Files 672 672 Lines 3756037564 +4 == + Hits3148331487 +4 Misses 6077 6077 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/api/experimental/endpoints.py](https://codecov.io/gh/apache/airflow/pull/6651/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvYXBpL2V4cGVyaW1lbnRhbC9lbmRwb2ludHMucHk=) | `88.71% <100%> (+0.23%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=footer). Last update [db4be19...ce2e499](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6651: [WIP][AIRFLOW-XXXX] Add info endpoint to experimental api
codecov-io commented on issue #6651: [WIP][AIRFLOW-] Add info endpoint to experimental api URL: https://github.com/apache/airflow/pull/6651#issuecomment-557893722 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=h1) Report > Merging [#6651](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/db4be19b80a65913edb5ab634226cf60ea8bffea?src=pr=desc) will **increase** coverage by `<.01%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6651 +/- ## == + Coverage 83.82% 83.82% +<.01% == Files 672 672 Lines 3756037564 +4 == + Hits3148331487 +4 Misses 6077 6077 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/api/experimental/endpoints.py](https://codecov.io/gh/apache/airflow/pull/6651/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvYXBpL2V4cGVyaW1lbnRhbC9lbmRwb2ludHMucHk=) | `88.71% <100%> (+0.23%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=footer). Last update [db4be19...ce2e499](https://codecov.io/gh/apache/airflow/pull/6651?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] aviemzur commented on issue #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id
aviemzur commented on issue #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id URL: https://github.com/apache/airflow/pull/6655#issuecomment-557892259 > @aviemzur can you fix error? > `airflow/contrib/hooks/emr_hook.py:48:111: E501 line too long (115 > 110 characters) ` Yup, just pushed a fix This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] OmerJog commented on issue #6652: [AIRFLOW-5548] [AIRFLOW-5550] REST API enhancement - dag info, task …
OmerJog commented on issue #6652: [AIRFLOW-5548] [AIRFLOW-5550] REST API enhancement - dag info, task … URL: https://github.com/apache/airflow/pull/6652#issuecomment-557891069 @msb217 can you fix style errors? ``` airflow/www/api/experimental/endpoints.py:177:1: E302 expected 2 blank lines, found 1 airflow/api/common/experimental/get_task.py:33:1: E302 expected 2 blank lines, found 1 airflow/api/common/experimental/get_task.py:33:46: E231 missing whitespace after ':' airflow/api/common/experimental/__init__.py:49:1: E302 expected 2 blank lines, found 1 ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] OmerJog commented on issue #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id
OmerJog commented on issue #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id URL: https://github.com/apache/airflow/pull/6655#issuecomment-557890912 @aviemzur can you fix error? `airflow/contrib/hooks/emr_hook.py:48:111: E501 line too long (115 > 110 characters) ` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] bolkedebruin commented on issue #6564: [AIRFLOW-5911] Simplify lineage API and improve robustness
bolkedebruin commented on issue #6564: [AIRFLOW-5911] Simplify lineage API and improve robustness URL: https://github.com/apache/airflow/pull/6564#issuecomment-557890511 Build errors (pylint) seem unrelated. See https://travis-ci.org/bolkedebruin/airflow/builds/616252609 for one that passed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] bolkedebruin commented on issue #6564: [AIRFLOW-5911] Simplify lineage API and improve robustness
bolkedebruin commented on issue #6564: [AIRFLOW-5911] Simplify lineage API and improve robustness URL: https://github.com/apache/airflow/pull/6564#issuecomment-557884955 @ashb @kaxil should be ready for review. Note that I will improve the docs when I have done the sub-sequent PRs This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6056) Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id
[
https://issues.apache.org/jira/browse/AIRFLOW-6056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981058#comment-16981058
]
ASF GitHub Bot commented on AIRFLOW-6056:
-
aviemzur commented on pull request #6655: [AIRFLOW-6056] Allow
EmrAddStepsOperator to accept job_flow_name as a…
URL: https://github.com/apache/airflow/pull/6655
…lternative to job_flow_id
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Allow EmrAddStepsOperator to accept job_flow_name as alternative to
> job_flow_id
> ---
>
> Key: AIRFLOW-6056
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6056
> Project: Apache Airflow
> Issue Type: New Feature
> Components: contrib
>Affects Versions: 1.10.6
>Reporter: Aviem Zur
>Assignee: Aviem Zur
>Priority: Major
>
> Allow EmrAddStepsOperator to accept job_flow_name as alternative to
> job_flow_id
> This is useful for when you have a persistent EMR cluster for testing or for
> when you have repeated scheduled jobs with a small interval between them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] aviemzur opened a new pull request #6655: [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as a…
aviemzur opened a new pull request #6655: [AIRFLOW-6056] Allow
EmrAddStepsOperator to accept job_flow_name as a…
URL: https://github.com/apache/airflow/pull/6655
…lternative to job_flow_id
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-XXX
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6635: [AIRFLOW-6033] Fix UI Crash at "Landing Times" when change task_id wi…
codecov-io edited a comment on issue #6635: [AIRFLOW-6033] Fix UI Crash at "Landing Times" when change task_id wi… URL: https://github.com/apache/airflow/pull/6635#issuecomment-557815829 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6635?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@b401409`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6635?src=pr=tree) ```diff @@Coverage Diff@@ ## master#6635 +/- ## = Coverage ? 83.82% = Files ? 672 Lines ?37561 Branches ?0 = Hits ?31484 Misses? 6077 Partials ?0 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6635?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/6635/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.6% <100%> (ø)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6635?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6635?src=pr=footer). Last update [b401409...29f2568](https://codecov.io/gh/apache/airflow/pull/6635?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (AIRFLOW-6056) Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id
Aviem Zur created AIRFLOW-6056: -- Summary: Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id Key: AIRFLOW-6056 URL: https://issues.apache.org/jira/browse/AIRFLOW-6056 Project: Apache Airflow Issue Type: New Feature Components: contrib Affects Versions: 1.10.6 Reporter: Aviem Zur Assignee: Aviem Zur Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id This is useful for when you have a persistent EMR cluster for testing or for when you have repeated scheduled jobs with a small interval between them. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] drexpp commented on issue #6635: [AIRFLOW-6033] Fix UI Crash at "Landing Times" when change task_id wi…
drexpp commented on issue #6635: [AIRFLOW-6033] Fix UI Crash at "Landing Times" when change task_id wi… URL: https://github.com/apache/airflow/pull/6635#issuecomment-557879705 > > The given error is quite weird > > ``` > > 44) FAIL: test_heartbeat_failed_fast (tests.jobs.test_local_task_job.TestLocalTaskJob) > > > >Traceback (most recent call last): > > /usr/local/lib/python3.7/unittest/mock.py line 1255 in patched > > return func(*args, **keywargs) > > tests/jobs/test_local_task_job.py line 161 in test_heartbeat_failed_fast > > self.assertGreaterEqual((time2 - time1).total_seconds(), job.heartrate) > >AssertionError: 1.997115 not greater than or equal to 2 > > ``` > > > > > > And I think it might not be related to my pull request. > > It is just a transient error. I have restarted the test, should fix it. Thanks for restarting it @kaxil it seems like everything went fine, thanks for your time. Then I will do the PR against the README.md thanks for the reference. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6055) Exponential Backoff in Sensors
[
https://issues.apache.org/jira/browse/AIRFLOW-6055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981037#comment-16981037
]
ASF GitHub Bot commented on AIRFLOW-6055:
-
msumit commented on pull request #6654: AIRFLOW-6055: Option for exponential
backoff in Sensors
URL: https://github.com/apache/airflow/pull/6654
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issue
https://issues.apache.org/jira/browse/AIRFLOW-6055
### Description
- [ ] Adding an option in sensors to do exponential backoff before poke or
schedule again.
### Tests
- [ ] Added a couple of unit tests around the new function
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Exponential Backoff in Sensors
> --
>
> Key: AIRFLOW-6055
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6055
> Project: Apache Airflow
> Issue Type: New Feature
> Components: operators
>Affects Versions: 2.0.0
>Reporter: Sumit Maheshwari
>Assignee: Sumit Maheshwari
>Priority: Major
>
> Like operators, there should be an option in Sensors as well to do
> exponential backoff.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] msumit opened a new pull request #6654: AIRFLOW-6055: Option for exponential backoff in Sensors
msumit opened a new pull request #6654: AIRFLOW-6055: Option for exponential
backoff in Sensors
URL: https://github.com/apache/airflow/pull/6654
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issue
https://issues.apache.org/jira/browse/AIRFLOW-6055
### Description
- [ ] Adding an option in sensors to do exponential backoff before poke or
schedule again.
### Tests
- [ ] Added a couple of unit tests around the new function
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-6055) Exponential Backoff in Sensors
Sumit Maheshwari created AIRFLOW-6055: - Summary: Exponential Backoff in Sensors Key: AIRFLOW-6055 URL: https://issues.apache.org/jira/browse/AIRFLOW-6055 Project: Apache Airflow Issue Type: New Feature Components: operators Affects Versions: 2.0.0 Reporter: Sumit Maheshwari Assignee: Sumit Maheshwari Like operators, there should be an option in Sensors as well to do exponential backoff. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] OmerJog commented on issue #5458: [AIRFLOW-4495] allow externally triggered dags to run for future 'Exe…
OmerJog commented on issue #5458: [AIRFLOW-4495] allow externally triggered dags to run for future 'Exe… URL: https://github.com/apache/airflow/pull/5458#issuecomment-557874004 @tooptoop4 one of the tests timeouted. no worries it doesn't meant anything. Just wait for review from one of the committers This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6650: [AIRFLOW-6051] Make DAG optional during displaying the log
codecov-io edited a comment on issue #6650: [AIRFLOW-6051] Make DAG optional during displaying the log URL: https://github.com/apache/airflow/pull/6650#issuecomment-557848169 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6650?src=pr=h1) Report > Merging [#6650](https://codecov.io/gh/apache/airflow/pull/6650?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3ac5270550eb0453c5e3f9477ad607f79828bc78?src=pr=desc) will **increase** coverage by `0.03%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6650?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6650 +/- ## == + Coverage 83.78% 83.81% +0.03% == Files 672 672 Lines 3756737561 -6 == + Hits3147631483 +7 + Misses 6091 6078 -13 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6650?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.6% <100%> (+0.01%)` | :arrow_up: | | [airflow/cli/commands/worker\_command.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvd29ya2VyX2NvbW1hbmQucHk=) | `38.09% <0%> (-1.44%)` | :arrow_down: | | [airflow/cli/commands/pool\_command.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvcG9vbF9jb21tYW5kLnB5) | `89.06% <0%> (-1.08%)` | :arrow_down: | | [airflow/cli/commands/task\_command.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvdGFza19jb21tYW5kLnB5) | `62.01% <0%> (-0.3%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.15% <0%> (-0.17%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `98.71% <0%> (+0.1%)` | :arrow_up: | | [airflow/models/dagbag.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnYmFnLnB5) | `87.29% <0%> (+0.4%)` | :arrow_up: | | [airflow/cli/commands/dag\_command.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGFnX2NvbW1hbmQucHk=) | `81.11% <0%> (+0.84%)` | :arrow_up: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `90% <0%> (+3.75%)` | :arrow_up: | | [airflow/kubernetes/pod\_runtime\_info\_env.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9ydW50aW1lX2luZm9fZW52LnB5) | `50% <0%> (+50%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6650?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6650?src=pr=footer). Last update [3ac5270...35969fe](https://codecov.io/gh/apache/airflow/pull/6650?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6650: [AIRFLOW-6051] Make DAG optional during displaying the log
codecov-io edited a comment on issue #6650: [AIRFLOW-6051] Make DAG optional during displaying the log URL: https://github.com/apache/airflow/pull/6650#issuecomment-557848169 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6650?src=pr=h1) Report > Merging [#6650](https://codecov.io/gh/apache/airflow/pull/6650?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3ac5270550eb0453c5e3f9477ad607f79828bc78?src=pr=desc) will **increase** coverage by `0.03%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6650?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6650 +/- ## == + Coverage 83.78% 83.81% +0.03% == Files 672 672 Lines 3756737561 -6 == + Hits3147631483 +7 + Misses 6091 6078 -13 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6650?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.6% <100%> (+0.01%)` | :arrow_up: | | [airflow/cli/commands/worker\_command.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvd29ya2VyX2NvbW1hbmQucHk=) | `38.09% <0%> (-1.44%)` | :arrow_down: | | [airflow/cli/commands/pool\_command.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvcG9vbF9jb21tYW5kLnB5) | `89.06% <0%> (-1.08%)` | :arrow_down: | | [airflow/cli/commands/task\_command.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvdGFza19jb21tYW5kLnB5) | `62.01% <0%> (-0.3%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.15% <0%> (-0.17%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `98.71% <0%> (+0.1%)` | :arrow_up: | | [airflow/models/dagbag.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnYmFnLnB5) | `87.29% <0%> (+0.4%)` | :arrow_up: | | [airflow/cli/commands/dag\_command.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGFnX2NvbW1hbmQucHk=) | `81.11% <0%> (+0.84%)` | :arrow_up: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `90% <0%> (+3.75%)` | :arrow_up: | | [airflow/kubernetes/pod\_runtime\_info\_env.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9ydW50aW1lX2luZm9fZW52LnB5) | `50% <0%> (+50%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6650?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6650?src=pr=footer). Last update [3ac5270...35969fe](https://codecov.io/gh/apache/airflow/pull/6650?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6650: [AIRFLOW-6051] Make DAG optional during displaying the log
codecov-io edited a comment on issue #6650: [AIRFLOW-6051] Make DAG optional during displaying the log URL: https://github.com/apache/airflow/pull/6650#issuecomment-557848169 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6650?src=pr=h1) Report > Merging [#6650](https://codecov.io/gh/apache/airflow/pull/6650?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/3ac5270550eb0453c5e3f9477ad607f79828bc78?src=pr=desc) will **increase** coverage by `0.03%`. > The diff coverage is `100%`. [](https://codecov.io/gh/apache/airflow/pull/6650?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6650 +/- ## == + Coverage 83.78% 83.81% +0.03% == Files 672 672 Lines 3756737561 -6 == + Hits3147631483 +7 + Misses 6091 6078 -13 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6650?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/www/views.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdmlld3MucHk=) | `76.6% <100%> (+0.01%)` | :arrow_up: | | [airflow/cli/commands/worker\_command.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvd29ya2VyX2NvbW1hbmQucHk=) | `38.09% <0%> (-1.44%)` | :arrow_down: | | [airflow/cli/commands/pool\_command.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvcG9vbF9jb21tYW5kLnB5) | `89.06% <0%> (-1.08%)` | :arrow_down: | | [airflow/cli/commands/task\_command.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvdGFza19jb21tYW5kLnB5) | `62.01% <0%> (-0.3%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.15% <0%> (-0.17%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `98.71% <0%> (+0.1%)` | :arrow_up: | | [airflow/models/dagbag.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvZGFnYmFnLnB5) | `87.29% <0%> (+0.4%)` | :arrow_up: | | [airflow/cli/commands/dag\_command.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGFnX2NvbW1hbmQucHk=) | `81.11% <0%> (+0.84%)` | :arrow_up: | | [airflow/jobs/local\_task\_job.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzL2xvY2FsX3Rhc2tfam9iLnB5) | `90% <0%> (+3.75%)` | :arrow_up: | | [airflow/kubernetes/pod\_runtime\_info\_env.py](https://codecov.io/gh/apache/airflow/pull/6650/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9ydW50aW1lX2luZm9fZW52LnB5) | `50% <0%> (+50%)` | :arrow_up: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6650?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6650?src=pr=footer). Last update [3ac5270...35969fe](https://codecov.io/gh/apache/airflow/pull/6650?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #6653: [AIRFLOW-6054] Add a command that starts the database consoles
codecov-io commented on issue #6653: [AIRFLOW-6054] Add a command that starts the database consoles URL: https://github.com/apache/airflow/pull/6653#issuecomment-557870434 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6653?src=pr=h1) Report > Merging [#6653](https://codecov.io/gh/apache/airflow/pull/6653?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/db4be19b80a65913edb5ab634226cf60ea8bffea?src=pr=desc) will **decrease** coverage by `0.41%`. > The diff coverage is `96%`. [](https://codecov.io/gh/apache/airflow/pull/6653?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6653 +/- ## == - Coverage 83.82% 83.41% -0.42% == Files 672 672 Lines 3756037584 +24 == - Hits3148331349 -134 - Misses 6077 6235 +158 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6653?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/bin/cli.py](https://codecov.io/gh/apache/airflow/pull/6653/diff?src=pr=tree#diff-YWlyZmxvdy9iaW4vY2xpLnB5) | `92.64% <ø> (ø)` | :arrow_up: | | [airflow/cli/commands/db\_command.py](https://codecov.io/gh/apache/airflow/pull/6653/diff?src=pr=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvZGJfY29tbWFuZC5weQ==) | `94.87% <96%> (+14.87%)` | :arrow_up: | | [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/6653/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0%> (-55.56%)` | :arrow_down: | | [airflow/executors/sequential\_executor.py](https://codecov.io/gh/apache/airflow/pull/6653/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvc2VxdWVudGlhbF9leGVjdXRvci5weQ==) | `47.61% <0%> (-52.39%)` | :arrow_down: | | [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/6653/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0%> (-47.06%)` | :arrow_down: | | [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/6653/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `45.25% <0%> (-46.72%)` | :arrow_down: | | [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/6653/diff?src=pr=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0%> (-23.53%)` | :arrow_down: | | [...rflow/contrib/operators/kubernetes\_pod\_operator.py](https://codecov.io/gh/apache/airflow/pull/6653/diff?src=pr=tree#diff-YWlyZmxvdy9jb250cmliL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZF9vcGVyYXRvci5weQ==) | `78.2% <0%> (-20.52%)` | :arrow_down: | | [airflow/utils/sqlalchemy.py](https://codecov.io/gh/apache/airflow/pull/6653/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zcWxhbGNoZW15LnB5) | `86.44% <0%> (-6.78%)` | :arrow_down: | | [airflow/executors/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/6653/diff?src=pr=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvX19pbml0X18ucHk=) | `63.26% <0%> (-4.09%)` | :arrow_down: | | ... and [3 more](https://codecov.io/gh/apache/airflow/pull/6653/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6653?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6653?src=pr=footer). Last update [db4be19...63dc850](https://codecov.io/gh/apache/airflow/pull/6653?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj edited a comment on issue #6644: [AIRFLOW-6047] Simplify the logging configuration template
mik-laj edited a comment on issue #6644: [AIRFLOW-6047] Simplify the logging configuration template URL: https://github.com/apache/airflow/pull/6644#issuecomment-557869288 I would dream that the whole configuration was expressed as Python code. Then in the HOME directory we would have settings.py with the following content: ``` from airflow.settings import * ``` And the airflow.settings file would contain default value definitions that could be overwritten in settings.py. ``` ENABLE_AWESOME_FEATURE = True ``` This would allow much greater freedom and create more complex objects, e.g. dictionaries. This is how Django is configured. https://docs.djangoproject.com/en/2.2/topics/settings/ https://github.com/joke2k/django-environ But all in all, if we delete the import from the first code and transfer it to the core, we will have your idea. so i agree. We can move this configuration file to the core and introduce another way to overwrite the settings. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj edited a comment on issue #6644: [AIRFLOW-6047] Simplify the logging configuration template
mik-laj edited a comment on issue #6644: [AIRFLOW-6047] Simplify the logging configuration template URL: https://github.com/apache/airflow/pull/6644#issuecomment-557869288 I would dream that the whole configuration was expressed as Python code. Then in the HOME directory we would have settings.py with the following content: ``` from airflow.settings import * ``` And the airflow.settings file would contain default value definitions that could be overwritten in settings.py. ``` ENABLE_AWESOME_FEATURE = True ``` This would allow much greater freedom and create more complex objects, e.g. dictionaries. This is how Django is configured. https://docs.djangoproject.com/en/2.2/topics/settings/ https://github.com/joke2k/django-environ But all in all, if we delete the import from the first code and transfer it to the core, we will have your idea. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #6644: [AIRFLOW-6047] Simplify the logging configuration template
mik-laj commented on issue #6644: [AIRFLOW-6047] Simplify the logging configuration template URL: https://github.com/apache/airflow/pull/6644#issuecomment-557869288 I would dream that the whole configuration was expressed as Python code. Then in the HOME directory we would have settings.py with the following content: ``` from airflow.settings import * ``` And the airflow.settings file would contain default value definitions that could be overwritten in settings.py. ``` ENABLE_AWESOME_FEATURE = True ``` This would allow much greater freedom and create more complex objects, e.g. dictionaries. This is how Django is configured. https://docs.djangoproject.com/en/2.2/topics/settings/ https://github.com/joke2k/django-environ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] nuclearpinguin closed pull request #6626: Test commit
nuclearpinguin closed pull request #6626: Test commit URL: https://github.com/apache/airflow/pull/6626 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-6054) Add a command that starts the database consoles
[
https://issues.apache.org/jira/browse/AIRFLOW-6054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16980997#comment-16980997
]
ASF GitHub Bot commented on AIRFLOW-6054:
-
mik-laj commented on pull request #6653: [AIRFLOW-6054] Add a command that
starts the database consoles
URL: https://github.com/apache/airflow/pull/6653
@potiuk This will be a very nice improvement for development environment
including Breeze.
---
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-6054
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Add a command that starts the database consoles
> ---
>
> Key: AIRFLOW-6054
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6054
> Project: Apache Airflow
> Issue Type: Bug
> Components: cli
>Affects Versions: 1.10.6
>Reporter: Kamil Bregula
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] mik-laj opened a new pull request #6653: [AIRFLOW-6054] Add a command that starts the database consoles
mik-laj opened a new pull request #6653: [AIRFLOW-6054] Add a command that
starts the database consoles
URL: https://github.com/apache/airflow/pull/6653
@potiuk This will be a very nice improvement for development environment
including Breeze.
---
Make sure you have checked _all_ steps below.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW/) issues and references
them in the PR title. For example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-6054
- In case you are fixing a typo in the documentation you can prepend your
commit with \[AIRFLOW-XXX\], code changes always need a Jira issue.
- In case you are proposing a fundamental code change, you need to create
an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)).
- In case you are adding a dependency, check if the license complies with
the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
### Description
- [ ] Here are some details about my PR, including screenshots of any UI
changes:
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Created] (AIRFLOW-6054) Add a command that starts the database consoles
Kamil Bregula created AIRFLOW-6054: -- Summary: Add a command that starts the database consoles Key: AIRFLOW-6054 URL: https://issues.apache.org/jira/browse/AIRFLOW-6054 Project: Apache Airflow Issue Type: Bug Components: cli Affects Versions: 1.10.6 Reporter: Kamil Bregula -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] codecov-io edited a comment on issue #6601: [AIRFLOW-6010] Remove cyclic imports and pylint disables
codecov-io edited a comment on issue #6601: [AIRFLOW-6010] Remove cyclic imports and pylint disables URL: https://github.com/apache/airflow/pull/6601#issuecomment-557836097 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6601?src=pr=h1) Report > Merging [#6601](https://codecov.io/gh/apache/airflow/pull/6601?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/db4be19b80a65913edb5ab634226cf60ea8bffea?src=pr=desc) will **increase** coverage by `0.01%`. > The diff coverage is `96.68%`. [](https://codecov.io/gh/apache/airflow/pull/6601?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6601 +/- ## == + Coverage 83.82% 83.83% +0.01% == Files 672 671 -1 Lines 3756037574 +14 == + Hits3148331500 +17 + Misses 6077 6074 -3 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6601?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/settings.py](https://codecov.io/gh/apache/airflow/pull/6601/diff?src=pr=tree#diff-YWlyZmxvdy9zZXR0aW5ncy5weQ==) | `88.81% <ø> (-0.16%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6601/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.15% <0%> (-0.33%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6601/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `92.75% <100%> (ø)` | :arrow_up: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6601/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.28% <100%> (+0.07%)` | :arrow_up: | | [airflow/www/utils.py](https://codecov.io/gh/apache/airflow/pull/6601/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdXRpbHMucHk=) | `80.19% <100%> (ø)` | :arrow_up: | | [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/6601/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `65% <100%> (ø)` | :arrow_up: | | [airflow/utils/helpers.py](https://codecov.io/gh/apache/airflow/pull/6601/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5) | `77.97% <100%> (ø)` | :arrow_up: | | [airflow/operators/bigquery\_to\_mysql.py](https://codecov.io/gh/apache/airflow/pull/6601/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvYmlncXVlcnlfdG9fbXlzcWwucHk=) | `74% <100%> (ø)` | :arrow_up: | | [...e/marketing\_platform/operators/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6601/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL21hcmtldGluZ19wbGF0Zm9ybS9vcGVyYXRvcnMvY2FtcGFpZ25fbWFuYWdlci5weQ==) | `91.73% <100%> (ø)` | :arrow_up: | | [airflow/api/common/experimental/mark\_tasks.py](https://codecov.io/gh/apache/airflow/pull/6601/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9tYXJrX3Rhc2tzLnB5) | `95.39% <100%> (+0.03%)` | :arrow_up: | | ... and [21 more](https://codecov.io/gh/apache/airflow/pull/6601/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6601?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6601?src=pr=footer). Last update [db4be19...0e6b5c4](https://codecov.io/gh/apache/airflow/pull/6601?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #6601: [AIRFLOW-6010] Remove cyclic imports and pylint disables
codecov-io edited a comment on issue #6601: [AIRFLOW-6010] Remove cyclic imports and pylint disables URL: https://github.com/apache/airflow/pull/6601#issuecomment-557836097 # [Codecov](https://codecov.io/gh/apache/airflow/pull/6601?src=pr=h1) Report > Merging [#6601](https://codecov.io/gh/apache/airflow/pull/6601?src=pr=desc) into [master](https://codecov.io/gh/apache/airflow/commit/db4be19b80a65913edb5ab634226cf60ea8bffea?src=pr=desc) will **increase** coverage by `0.01%`. > The diff coverage is `96.68%`. [](https://codecov.io/gh/apache/airflow/pull/6601?src=pr=tree) ```diff @@Coverage Diff @@ ## master#6601 +/- ## == + Coverage 83.82% 83.83% +0.01% == Files 672 671 -1 Lines 3756037574 +14 == + Hits3148331500 +17 + Misses 6077 6074 -3 ``` | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/6601?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/settings.py](https://codecov.io/gh/apache/airflow/pull/6601/diff?src=pr=tree#diff-YWlyZmxvdy9zZXR0aW5ncy5weQ==) | `88.81% <ø> (-0.16%)` | :arrow_down: | | [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/6601/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `58.15% <0%> (-0.33%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/airflow/pull/6601/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `92.75% <100%> (ø)` | :arrow_up: | | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/6601/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.28% <100%> (+0.07%)` | :arrow_up: | | [airflow/www/utils.py](https://codecov.io/gh/apache/airflow/pull/6601/diff?src=pr=tree#diff-YWlyZmxvdy93d3cvdXRpbHMucHk=) | `80.19% <100%> (ø)` | :arrow_up: | | [airflow/models/connection.py](https://codecov.io/gh/apache/airflow/pull/6601/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMvY29ubmVjdGlvbi5weQ==) | `65% <100%> (ø)` | :arrow_up: | | [airflow/utils/helpers.py](https://codecov.io/gh/apache/airflow/pull/6601/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5) | `77.97% <100%> (ø)` | :arrow_up: | | [airflow/operators/bigquery\_to\_mysql.py](https://codecov.io/gh/apache/airflow/pull/6601/diff?src=pr=tree#diff-YWlyZmxvdy9vcGVyYXRvcnMvYmlncXVlcnlfdG9fbXlzcWwucHk=) | `74% <100%> (ø)` | :arrow_up: | | [...e/marketing\_platform/operators/campaign\_manager.py](https://codecov.io/gh/apache/airflow/pull/6601/diff?src=pr=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL21hcmtldGluZ19wbGF0Zm9ybS9vcGVyYXRvcnMvY2FtcGFpZ25fbWFuYWdlci5weQ==) | `91.73% <100%> (ø)` | :arrow_up: | | [airflow/api/common/experimental/mark\_tasks.py](https://codecov.io/gh/apache/airflow/pull/6601/diff?src=pr=tree#diff-YWlyZmxvdy9hcGkvY29tbW9uL2V4cGVyaW1lbnRhbC9tYXJrX3Rhc2tzLnB5) | `95.39% <100%> (+0.03%)` | :arrow_up: | | ... and [21 more](https://codecov.io/gh/apache/airflow/pull/6601/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/6601?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/6601?src=pr=footer). Last update [db4be19...0e6b5c4](https://codecov.io/gh/apache/airflow/pull/6601?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
