[GitHub] [airflow] Khrol commented on pull request #7622: [AIRFLOW-6983] Use SingletonThreadPool for database communication

2020-05-20 Thread GitBox
Khrol commented on pull request #7622: URL: https://github.com/apache/airflow/pull/7622#issuecomment-631898187 https://docs.sqlalchemy.org/en/13/core/pooling.html#sqlalchemy.pool.SingletonThreadPool ``` SingletonThreadPool may be improved in a future release, however in its

[jira] [Commented] (AIRFLOW-6983) Disabled connection pool in CLI might be harmful

2020-05-20 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112822#comment-17112822 ] ASF GitHub Bot commented on AIRFLOW-6983: - Khrol commented on pull request #7622: URL:

[jira] [Commented] (AIRFLOW-6983) Disabled connection pool in CLI might be harmful

2020-05-20 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112821#comment-17112821 ] ASF GitHub Bot commented on AIRFLOW-6983: - Khrol commented on a change in pull request #7622:

[GitHub] [airflow] Khrol commented on a change in pull request #7622: [AIRFLOW-6983] Use SingletonThreadPool for database communication

2020-05-20 Thread GitBox
Khrol commented on a change in pull request #7622: URL: https://github.com/apache/airflow/pull/7622#discussion_r428460044 ## File path: airflow/settings.py ## @@ -148,18 +148,6 @@ def configure_orm(disable_connection_pool=False): # 0 means no limit, which could lead

[GitHub] [airflow] yuqian90 commented on pull request #8867: Fixed BaseSensorOperator to make respect the trigger_rule in downstream tasks, when setting soft_fail="True"

2020-05-20 Thread GitBox
yuqian90 commented on pull request #8867: URL: https://github.com/apache/airflow/pull/8867#issuecomment-631890833 > On a side note... > If you know of a documentation file that needs to be modified, please let me know. I think the document you need to update is UPDATING.md. It

[GitHub] [airflow] potiuk edited a comment on pull request #8938: Hive/Hadoop minicluster needs JDK8 and JAVA_HOME to work

2020-05-20 Thread GitBox
potiuk edited a comment on pull request #8938: URL: https://github.com/apache/airflow/pull/8938#issuecomment-631888486 Thanks @ashb ! I merged it now - even though one of the tests failed (but It looked like a transient error). Once the master build for this one succeeds, the case

[GitHub] [airflow] potiuk edited a comment on pull request #8938: Hive/Hadoop minicluster needs JDK8 and JAVA_HOME to work

2020-05-20 Thread GitBox
potiuk edited a comment on pull request #8938: URL: https://github.com/apache/airflow/pull/8938#issuecomment-631888486 Thanks @ashb ! I merged it now - even though one of the tests failed (but It looked like a transient error). Once the master build for this one succeeds, the case

[GitHub] [airflow] potiuk commented on pull request #8938: Hive/Hadoop minicluster needs JDK8 and JAVA_HOME to work

2020-05-20 Thread GitBox
potiuk commented on pull request #8938: URL: https://github.com/apache/airflow/pull/8938#issuecomment-631888486 Thanks @ashb ! I merged it now - even though one of the tests failed (but It looked like a transient error). Once the master build for this one succeeds, the case in the

[airflow] branch master updated (12c22e0 -> 8476c1e)

2020-05-20 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 12c22e0 Added Greytip to Airflow Users list (#8887) add 8476c1e Hive/Hadoop minicluster needs JDK8 and

[GitHub] [airflow] potiuk merged pull request #8938: Hive/Hadoop minicluster needs JDK8 and JAVA_HOME to work

2020-05-20 Thread GitBox
potiuk merged pull request #8938: URL: https://github.com/apache/airflow/pull/8938 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] potiuk edited a comment on pull request #8889: Support custom Variable implementation

2020-05-20 Thread GitBox
potiuk edited a comment on pull request #8889: URL: https://github.com/apache/airflow/pull/8889#issuecomment-631885153 I think maybe it's the right time to raise that as a separate thread on the devlist so that other people can state their opinion as well? I agree with @ashb and

[GitHub] [airflow] potiuk commented on pull request #8889: Support custom Variable implementation

2020-05-20 Thread GitBox
potiuk commented on pull request #8889: URL: https://github.com/apache/airflow/pull/8889#issuecomment-631885153 I think maybe it's the right time to raise that as a separate thread on the devlist so that other people can state their opinion as well? I agree with @ashb and @kaxil

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #8942: Add SQL Branch Operator #8525

2020-05-20 Thread GitBox
boring-cyborg[bot] commented on pull request #8942: URL: https://github.com/apache/airflow/pull/8942#issuecomment-631884466 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our

[GitHub] [airflow] samuelkhtu opened a new pull request #8942: Add SQL Branch Operator #8525

2020-05-20 Thread GitBox
samuelkhtu opened a new pull request #8942: URL: https://github.com/apache/airflow/pull/8942 SQL Branch Operator allow user to execute a SQL query in any supported backend to decide which branch of the DAG to follow. The SQL branch operator expects SQL query to return any of the

[GitHub] [airflow] samuelkhtu commented on issue #8525: SQLBranchOperator

2020-05-20 Thread GitBox
samuelkhtu commented on issue #8525: URL: https://github.com/apache/airflow/issues/8525#issuecomment-631871536 Code and test completed for this item. I am working on the PR. This is an automated message from the Apache Git

[GitHub] [airflow] ryanyuan commented on a change in pull request #8575: [AIRFLOW-6981] Move Google Cloud Build from Discovery API to Python Library

2020-05-20 Thread GitBox
ryanyuan commented on a change in pull request #8575: URL: https://github.com/apache/airflow/pull/8575#discussion_r428403561 ## File path: airflow/providers/google/cloud/operators/cloud_build.py ## @@ -15,186 +15,678 @@ # KIND, either express or implied. See the License for

[jira] [Commented] (AIRFLOW-6981) Move Google Cloud Build from Discovery API to Python Library

2020-05-20 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112717#comment-17112717 ] ASF GitHub Bot commented on AIRFLOW-6981: - ryanyuan commented on a change in pull request

[GitHub] [airflow] chrismclennon opened a new pull request #8941: Tree view tooltip to show next execution date

2020-05-20 Thread GitBox
chrismclennon opened a new pull request #8941: URL: https://github.com/apache/airflow/pull/8941 This proposed change modifies the tooltip on the Tree View to include "Next Execution Date". I've found that Airflow scheduling work at the end of a schedule interval is a common source of

[GitHub] [airflow] thesuperzapper commented on pull request #8777: Add Production Helm chart support

2020-05-20 Thread GitBox
thesuperzapper commented on pull request #8777: URL: https://github.com/apache/airflow/pull/8777#issuecomment-631823022 @schnie I have released v7.0.0 of the `stable/airflow` chart, [see here](https://github.com/helm/charts/tree/master/stable/airflow). There definitely are a few

[jira] [Commented] (AIRFLOW-249) Refactor the SLA mechanism

2020-05-20 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112691#comment-17112691 ] ASF GitHub Bot commented on AIRFLOW-249: houqp commented on a change in pull request #8545: URL:

[GitHub] [airflow] houqp commented on a change in pull request #8545: [AIRFLOW-249] Refactor the SLA mechanism (Continuation from #3584 )

2020-05-20 Thread GitBox
houqp commented on a change in pull request #8545: URL: https://github.com/apache/airflow/pull/8545#discussion_r428389970 ## File path: airflow/models/dagrun.py ## @@ -491,6 +491,25 @@ def is_backfill(self):

[GitHub] [airflow] mik-laj commented on pull request #8940: Remove side-effect of session in FAB

2020-05-20 Thread GitBox
mik-laj commented on pull request #8940: URL: https://github.com/apache/airflow/pull/8940#issuecomment-631815639 I need this change to integrate connexion with Airflow. This is an automated message from the Apache Git

[GitHub] [airflow] kaxil merged pull request #8887: added greytip to official company list

2020-05-20 Thread GitBox
kaxil merged pull request #8887: URL: https://github.com/apache/airflow/pull/8887 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[airflow] branch master updated: Added Greytip to Airflow Users list (#8887)

2020-05-20 Thread kaxilnaik
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/master by this push: new 12c22e0 Added Greytip to Airflow Users

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #8887: added greytip to official company list

2020-05-20 Thread GitBox
boring-cyborg[bot] commented on pull request #8887: URL: https://github.com/apache/airflow/pull/8887#issuecomment-631809699 Awesome work, congrats on your first merged pull request! This is an automated message from the

[airflow] branch master updated (51d9557 -> c6224e2)

2020-05-20 Thread kaxilnaik
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airflow.git. from 51d9557 Re-run all tests when Dockerfile or Github worflow change (#8924) add c6224e2 Remove unused

[GitHub] [airflow] kaxil merged pull request #8935: Remove unused SchedulerJob.max_threads argument

2020-05-20 Thread GitBox
kaxil merged pull request #8935: URL: https://github.com/apache/airflow/pull/8935 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[jira] [Commented] (AIRFLOW-249) Refactor the SLA mechanism

2020-05-20 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112677#comment-17112677 ] ASF GitHub Bot commented on AIRFLOW-249: seanxwzhang commented on a change in pull request

[GitHub] [airflow] seanxwzhang commented on a change in pull request #8545: [AIRFLOW-249] Refactor the SLA mechanism (Continuation from #3584 )

2020-05-20 Thread GitBox
seanxwzhang commented on a change in pull request #8545: URL: https://github.com/apache/airflow/pull/8545#discussion_r428375456 ## File path: airflow/models/dagrun.py ## @@ -491,6 +491,25 @@ def is_backfill(self):

[GitHub] [airflow] mik-laj commented on a change in pull request #8940: Remove side-effect of session in FAB

2020-05-20 Thread GitBox
mik-laj commented on a change in pull request #8940: URL: https://github.com/apache/airflow/pull/8940#discussion_r428373704 ## File path: tests/www/test_views.py ## @@ -428,7 +428,7 @@ def prepare_dagruns(self): state=State.RUNNING) def test_index(self): -

[GitHub] [airflow] mik-laj commented on a change in pull request #8940: Remove side-effect of session in FAB

2020-05-20 Thread GitBox
mik-laj commented on a change in pull request #8940: URL: https://github.com/apache/airflow/pull/8940#discussion_r428373315 ## File path: airflow/www/app.py ## @@ -46,7 +46,7 @@ log = logging.getLogger(__name__) -def create_app(config=None, session=None, testing=False,

[GitHub] [airflow] mik-laj commented on a change in pull request #8940: Remove side-effect of session in FAB

2020-05-20 Thread GitBox
mik-laj commented on a change in pull request #8940: URL: https://github.com/apache/airflow/pull/8940#discussion_r428372980 ## File path: airflow/www/app.py ## @@ -272,10 +282,6 @@ def apply_caching(response): response.headers["X-Frame-Options"] = "DENY"

[GitHub] [airflow] mik-laj commented on a change in pull request #8940: Remove side-effect of session in FAB

2020-05-20 Thread GitBox
mik-laj commented on a change in pull request #8940: URL: https://github.com/apache/airflow/pull/8940#discussion_r428372980 ## File path: airflow/www/app.py ## @@ -272,10 +282,6 @@ def apply_caching(response): response.headers["X-Frame-Options"] = "DENY"

[GitHub] [airflow] mik-laj commented on a change in pull request #8940: Remove side-effect of session in FAB

2020-05-20 Thread GitBox
mik-laj commented on a change in pull request #8940: URL: https://github.com/apache/airflow/pull/8940#discussion_r428372645 ## File path: airflow/www/app.py ## @@ -95,9 +96,18 @@ def create_app(config=None, session=None, testing=False, app_name="Airflow"):

[GitHub] [airflow] mik-laj commented on a change in pull request #8940: Remove side-effect of session in FAB

2020-05-20 Thread GitBox
mik-laj commented on a change in pull request #8940: URL: https://github.com/apache/airflow/pull/8940#discussion_r428370876 ## File path: airflow/www/app.py ## @@ -70,8 +70,9 @@ def create_app(config=None, session=None, testing=False, app_name="Airflow"):

[GitHub] [airflow] mik-laj opened a new pull request #8940: Remove side-effect of session in FAB

2020-05-20 Thread GitBox
mik-laj opened a new pull request #8940: URL: https://github.com/apache/airflow/pull/8940 --- Make sure to mark the boxes below before creating PR: [x] - [X] Description above provides context of the change - [X] Unit tests coverage for changes (not needed for documentation

[GitHub] [airflow] boring-cyborg[bot] commented on issue #8939: next_execution_date is not a pendulum instance

2020-05-20 Thread GitBox
boring-cyborg[bot] commented on issue #8939: URL: https://github.com/apache/airflow/issues/8939#issuecomment-631787061 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] danly opened a new issue #8939: next_execution_date is not a pendulum instance

2020-05-20 Thread GitBox
danly opened a new issue #8939: URL: https://github.com/apache/airflow/issues/8939 **Apache Airflow version**: 1.10.2 **What happened**: [According to the docs](https://airflow.apache.org/docs/stable/macros-ref.html) `next_execution_date` is a pendulum instance: > the

[GitHub] [airflow] snazzyfox commented on issue #8933: pyhive is installed without Hive dependencies

2020-05-20 Thread GitBox
snazzyfox commented on issue #8933: URL: https://github.com/apache/airflow/issues/8933#issuecomment-631786295 Yep, pyhive supports both hive and presto, so depending on the user's needs either `pyhive[hive]` or `pyhive[presto]` is installed. Since airflow is just installing `pyhive`

[GitHub] [airflow] zachliu commented on issue #8915: CSRF configuration is missing the WTF_ prefix

2020-05-20 Thread GitBox
zachliu commented on issue #8915: URL: https://github.com/apache/airflow/issues/8915#issuecomment-63137 > I'm not quite sure what you are asking us to do here -- the `webserver_config.py` that airflow generates does not set a time limit, `CSRF_TIME_LIMIT` or `WTF_CSRF_TIME_LIMIT`.

[GitHub] [airflow] ashb commented on issue #8717: DagRuns page renders the state column with artifacts

2020-05-20 Thread GitBox
ashb commented on issue #8717: URL: https://github.com/apache/airflow/issues/8717#issuecomment-631770256 This appears to a bug in our (soon to be remove) old UI. The easy fix for now is to switch to the RBAC UI by enabling it in your config files. We are only doing bug fixes against the

[GitHub] [airflow] ashb commented on issue #8484: Airflow 1.10.7+ suppresses Operator logs

2020-05-20 Thread GitBox
ashb commented on issue #8484: URL: https://github.com/apache/airflow/issues/8484#issuecomment-631768825 Is anyone able to make a clean, stand-along reproduction case that shows this please? (Ideally in the form of a docker image I can pull, or a git repo we can clone)

[GitHub] [airflow] dimberman edited a comment on pull request #8932: K8s tests on GitHub actions

2020-05-20 Thread GitBox
dimberman edited a comment on pull request #8932: URL: https://github.com/apache/airflow/pull/8932#issuecomment-631768481 @potiuk @ashb I'm starting to wonder if our image has gotten so big KinD can't even load it ``` ERROR: command "docker save -o

[GitHub] [airflow] dimberman commented on pull request #8932: K8s tests on GitHub actions

2020-05-20 Thread GitBox
dimberman commented on pull request #8932: URL: https://github.com/apache/airflow/pull/8932#issuecomment-631768481 @potiuk @ashb I'm starting to wonder if our image has gotten so big KinD can't even load it ``` ERROR: command "docker save -o /tmp/image-tar606047533/image.tar

[GitHub] [airflow] ashb closed issue #8753: Airflow suddenly died in all the instances

2020-05-20 Thread GitBox
ashb closed issue #8753: URL: https://github.com/apache/airflow/issues/8753 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] ashb commented on issue #8753: Airflow suddenly died in all the instances

2020-05-20 Thread GitBox
ashb commented on issue #8753: URL: https://github.com/apache/airflow/issues/8753#issuecomment-631768132 There is nothing we can do to debug this with so little information, sorry. This is an automated message from the

[GitHub] [airflow] ashb commented on issue #8779: Webserver OOM killed by Kubernetes when using KubernetesExecutor

2020-05-20 Thread GitBox
ashb commented on issue #8779: URL: https://github.com/apache/airflow/issues/8779#issuecomment-631767373 Are you using the "Run" button in the webserver by any chance? This is an automated message from the Apache Git

[GitHub] [airflow] ashb closed issue #8861: HttpHook.run method doesn't convert dict in 'data' parameter to json object

2020-05-20 Thread GitBox
ashb closed issue #8861: URL: https://github.com/apache/airflow/issues/8861 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] ashb commented on issue #8861: HttpHook.run method doesn't convert dict in 'data' parameter to json object

2020-05-20 Thread GitBox
ashb commented on issue #8861: URL: https://github.com/apache/airflow/issues/8861#issuecomment-631766490 I don't think anything needs doing, the hook already supports extra arguments

[GitHub] [airflow] ashb closed issue #8915: CSRF configuration is missing the WTF_ prefix

2020-05-20 Thread GitBox
ashb closed issue #8915: URL: https://github.com/apache/airflow/issues/8915 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] ashb commented on issue #8915: CSRF configuration is missing the WTF_ prefix

2020-05-20 Thread GitBox
ashb commented on issue #8915: URL: https://github.com/apache/airflow/issues/8915#issuecomment-631764412 I'm not quite sure what you are asking us to do here -- the `webserver_config.py` that airflow generates does not set a time limit, `CSRF_TIME_LIMIT` or `WTF_CSRF_TIME_LIMIT`.

[GitHub] [airflow] ashb commented on issue #8907: Airflow web UI is slow

2020-05-20 Thread GitBox
ashb commented on issue #8907: URL: https://github.com/apache/airflow/issues/8907#issuecomment-631763397 @sylr A flat 5s latency sounds odd, and isn't something Airflow itself is doing, so there's something environmental going on here. Things to check: if you run it locally, do you

[GitHub] [airflow] ashb commented on issue #8933: pyhive is installed without Hive dependencies

2020-05-20 Thread GitBox
ashb commented on issue #8933: URL: https://github.com/apache/airflow/issues/8933#issuecomment-631762509 Wait, pyhive has an extra called hive? :exploding_head: > I know this will be a much bigger conversation, but maybe it's worth it to consider testing operators only with the

[GitHub] [airflow] ashb closed pull request #8929: Add JAVA_HOME env var back to CI Docker image

2020-05-20 Thread GitBox
ashb closed pull request #8929: URL: https://github.com/apache/airflow/pull/8929 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] ashb commented on pull request #8929: Add JAVA_HOME env var back to CI Docker image

2020-05-20 Thread GitBox
ashb commented on pull request #8929: URL: https://github.com/apache/airflow/pull/8929#issuecomment-631753751 Closing in favour of #8938 This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [airflow] ashb opened a new pull request #8938: Hive/Hadoop minicluster needs JDK8 and JAVA_HOME to work

2020-05-20 Thread GitBox
ashb opened a new pull request #8938: URL: https://github.com/apache/airflow/pull/8938 Debian Buster only ships with a JDK11, and Hive/Hadoop fails in odd, hard to debug ways (complains about metastore not being initalized, possibly related to the class loader issues.) Until we

[GitHub] [airflow] ashb commented on pull request #8929: Add JAVA_HOME env var back to CI Docker image

2020-05-20 Thread GitBox
ashb commented on pull request #8929: URL: https://github.com/apache/airflow/pull/8929#issuecomment-631751289 Think I've got a fix -- using https://adoptopenjdk.net/index.html instead. This is an automated message from the

[jira] [Commented] (AIRFLOW-249) Refactor the SLA mechanism

2020-05-20 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112631#comment-17112631 ] ASF GitHub Bot commented on AIRFLOW-249: houqp commented on a change in pull request #8545: URL:

[GitHub] [airflow] houqp commented on a change in pull request #8545: [AIRFLOW-249] Refactor the SLA mechanism (Continuation from #3584 )

2020-05-20 Thread GitBox
houqp commented on a change in pull request #8545: URL: https://github.com/apache/airflow/pull/8545#discussion_r428324933 ## File path: airflow/models/dagrun.py ## @@ -491,6 +491,25 @@ def is_backfill(self):

[GitHub] [airflow] RosterIn opened a new issue #8937: change from zdesk to zenpy in ZendeskHook

2020-05-20 Thread GitBox
RosterIn opened a new issue #8937: URL: https://github.com/apache/airflow/issues/8937 **Description** [zdesk ](https://github.com/fprimex/zdesk) hasn't been updated in two years. there is a better maintained package [zenpy](https://github.com/facetoe/zenpy)

[GitHub] [airflow] ashb commented on pull request #8929: Add JAVA_HOME env var back to CI Docker image

2020-05-20 Thread GitBox
ashb commented on pull request #8929: URL: https://github.com/apache/airflow/pull/8929#issuecomment-631733162 This is failing now because Hive isn't creating (or has corrupt) meta store. But that doesn't _seem_ to be related to JDK vs JRE. Confused

[GitHub] [airflow] Acehaidrey closed pull request #8680: Add metric for start/end task run

2020-05-20 Thread GitBox
Acehaidrey closed pull request #8680: URL: https://github.com/apache/airflow/pull/8680 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [airflow] szczeles opened a new pull request #8936: Fixed S3FileTransformOperator to support S3 Select transformation only

2020-05-20 Thread GitBox
szczeles opened a new pull request #8936: URL: https://github.com/apache/airflow/pull/8936 Documentation for S3FileTransformOperator states that users can skip transformation script if S3 Select experession is specified, but in this case the created file is always zero bytes long.

[GitHub] [airflow] kaxil opened a new pull request #8935: Remove unused SchedulerJob.max_threads argument

2020-05-20 Thread GitBox
kaxil opened a new pull request #8935: URL: https://github.com/apache/airflow/pull/8935 SchedulerJob.max_threads is not used anywhere. --- Make sure to mark the boxes below before creating PR: [x] - [x] Description above provides context of the change - [x] Unit tests

[GitHub] [airflow] kaxil opened a new pull request #8934: Fix DagRun Prefix for Performance script

2020-05-20 Thread GitBox
kaxil opened a new pull request #8934: URL: https://github.com/apache/airflow/pull/8934 `DagRun.ID_PREFIX` no longer exists and is moved to `airflow.utils.types.DagRunType` enum. I have hardcoded `scheduled__` so that I can use it for 1.10.* too --- Make sure to mark the

[GitHub] [airflow] snazzyfox opened a new issue #8933: pyhive is installed without Hive dependencies

2020-05-20 Thread GitBox
snazzyfox opened a new issue #8933: URL: https://github.com/apache/airflow/issues/8933 **Apache Airflow version**: 1.10.10 (appears to also affect master) **What happened**: When airflow is installed with Hive support using `apache-airflow[hive]`, using HiveServer2Hook

[GitHub] [airflow] kaxil commented on a change in pull request #8852: Add task instance mutation hook

2020-05-20 Thread GitBox
kaxil commented on a change in pull request #8852: URL: https://github.com/apache/airflow/pull/8852#discussion_r428282991 ## File path: tests/models/test_dagrun.py ## @@ -564,6 +565,27 @@ def with_all_tasks_removed(dag): flaky_ti.refresh_from_db()

[GitHub] [airflow] kaxil commented on a change in pull request #8852: Add task instance mutation hook

2020-05-20 Thread GitBox
kaxil commented on a change in pull request #8852: URL: https://github.com/apache/airflow/pull/8852#discussion_r428282635 ## File path: airflow/models/dagrun.py ## @@ -431,6 +432,7 @@ def verify_integrity(self, session=None): # check for removed or restored tasks

[GitHub] [airflow] ashb commented on pull request #8929: Add JAVA_HOME env var back to CI Docker image

2020-05-20 Thread GitBox
ashb commented on pull request #8929: URL: https://github.com/apache/airflow/pull/8929#issuecomment-631693292 Not sure I can fix it, maybe we change default-jre-headless to default-jdk-headless in Dockerfile.ci Could you try that @dimberman ?

[jira] [Commented] (AIRFLOW-249) Refactor the SLA mechanism

2020-05-20 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112569#comment-17112569 ] ASF GitHub Bot commented on AIRFLOW-249: seanxwzhang commented on pull request #8545: URL:

[GitHub] [airflow] seanxwzhang commented on pull request #8545: [AIRFLOW-249] Refactor the SLA mechanism (Continuation from #3584 )

2020-05-20 Thread GitBox
seanxwzhang commented on pull request #8545: URL: https://github.com/apache/airflow/pull/8545#issuecomment-631691752 Made requested changes, please take a look again :) @houqp @BasPH thanks This is an automated message from

[GitHub] [airflow] chrismclennon edited a comment on issue #7935: scheduler gets stuck without a trace

2020-05-20 Thread GitBox
chrismclennon edited a comment on issue #7935: URL: https://github.com/apache/airflow/issues/7935#issuecomment-631684732 I see a similar issue on 1.10.9 where the scheduler runs fine on start but typically after 10 to 15 days the CPU utilization actually drops to near 0%. The scheduler

[GitHub] [airflow] milton0825 edited a comment on pull request #8889: Support custom Variable implementation

2020-05-20 Thread GitBox
milton0825 edited a comment on pull request #8889: URL: https://github.com/apache/airflow/pull/8889#issuecomment-631680187 Other use cases might be that we can replace the variable backend with other non-vault storage like S3 and support both read/write. It seems like `Variable` is

[GitHub] [airflow] chrismclennon commented on issue #7935: scheduler gets stuck without a trace

2020-05-20 Thread GitBox
chrismclennon commented on issue #7935: URL: https://github.com/apache/airflow/issues/7935#issuecomment-631684732 I see a similar issue on 1.10.9 where the scheduler runs fine on start but typically after ~10~15 days the CPU utilization actually drops to near 0%. The scheduler health

[GitHub] [airflow] milton0825 commented on a change in pull request #8852: Add task instance mutation hook

2020-05-20 Thread GitBox
milton0825 commented on a change in pull request #8852: URL: https://github.com/apache/airflow/pull/8852#discussion_r428262018 ## File path: airflow/models/dagrun.py ## @@ -431,6 +432,7 @@ def verify_integrity(self, session=None): # check for removed or restored tasks

[GitHub] [airflow] milton0825 edited a comment on pull request #8889: Support custom Variable implementation

2020-05-20 Thread GitBox
milton0825 edited a comment on pull request #8889: URL: https://github.com/apache/airflow/pull/8889#issuecomment-631680187 Other use cases might be that we can replace the variable backend with other storage like S3 and support both read/write. It seems like `Variable` is now

[GitHub] [airflow] milton0825 edited a comment on pull request #8889: Support custom Variable implementation

2020-05-20 Thread GitBox
milton0825 edited a comment on pull request #8889: URL: https://github.com/apache/airflow/pull/8889#issuecomment-631680187 Other use cases might be that we can replace the variable backend with other storage like S3 and support both read/write. It seems like `Variable` is now

[GitHub] [airflow] dimberman commented on pull request #8929: Add JAVA_HOME env var back to CI Docker image

2020-05-20 Thread GitBox
dimberman commented on pull request #8929: URL: https://github.com/apache/airflow/pull/8929#issuecomment-631680458 @ashb yup, was gonna mention it tonight (didn't want to bother you if you were off for the day) This is an

[GitHub] [airflow] milton0825 commented on pull request #8889: Support custom Variable implementation

2020-05-20 Thread GitBox
milton0825 commented on pull request #8889: URL: https://github.com/apache/airflow/pull/8889#issuecomment-631680187 Other use cases might be that we can replace the variable backend with other storage like S3 and support both read/write. It seems like `Variable` is now tightly

[GitHub] [airflow] ashb commented on pull request #8929: Add JAVA_HOME env var back to CI Docker image

2020-05-20 Thread GitBox
ashb commented on pull request #8929: URL: https://github.com/apache/airflow/pull/8929#issuecomment-631679851 Damn This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [airflow] dimberman opened a new pull request #8932: K8s tests on GitHub actions

2020-05-20 Thread GitBox
dimberman opened a new pull request #8932: URL: https://github.com/apache/airflow/pull/8932 --- Make sure to mark the boxes below before creating PR: [x] - [ ] Description above provides context of the change - [ ] Unit tests coverage for changes (not needed for documentation

[GitHub] [airflow] JeffryMAC opened a new issue #8931: Graph to see DAG run duration metric

2020-05-20 Thread GitBox
JeffryMAC opened a new issue #8931: URL: https://github.com/apache/airflow/issues/8931 **Description** airflow has many graph showing insights on task level (task duration, task tries, landing times, gantt) yet there are no graphs that show metrics on graph level I

[GitHub] [airflow] kaxil commented on a change in pull request #8852: Add task instance mutation hook

2020-05-20 Thread GitBox
kaxil commented on a change in pull request #8852: URL: https://github.com/apache/airflow/pull/8852#discussion_r428251168 ## File path: airflow/models/dagrun.py ## @@ -431,6 +432,7 @@ def verify_integrity(self, session=None): # check for removed or restored tasks

[GitHub] [airflow] milton0825 commented on a change in pull request #8852: Add task instance mutation hook

2020-05-20 Thread GitBox
milton0825 commented on a change in pull request #8852: URL: https://github.com/apache/airflow/pull/8852#discussion_r428248777 ## File path: airflow/models/dagrun.py ## @@ -431,6 +432,7 @@ def verify_integrity(self, session=None): # check for removed or restored tasks

[jira] [Commented] (AIRFLOW-5615) BaseJob subclasses shouldn't implement own heartbeat logic

2020-05-20 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112532#comment-17112532 ] ASF GitHub Bot commented on AIRFLOW-5615: - houqp commented on a change in pull request #6311:

[GitHub] [airflow] houqp commented on a change in pull request #6311: [AIRFLOW-5615] Reduce duplicated logic around job heartbeating

2020-05-20 Thread GitBox
houqp commented on a change in pull request #6311: URL: https://github.com/apache/airflow/pull/6311#discussion_r428236112 ## File path: airflow/jobs/base_job.py ## @@ -147,7 +147,7 @@ def heartbeat_callback(self, session=None): Callback that is called during

[GitHub] [airflow] randr97 commented on a change in pull request #8895: Add Delete/Create S3 bucket operators

2020-05-20 Thread GitBox
randr97 commented on a change in pull request #8895: URL: https://github.com/apache/airflow/pull/8895#discussion_r428229819 ## File path: airflow/providers/amazon/aws/operators/s3_bucket.py ## @@ -0,0 +1,119 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [airflow] maxcountryman commented on issue #8243: Sentry tags appear to be missing

2020-05-20 Thread GitBox
maxcountryman commented on issue #8243: URL: https://github.com/apache/airflow/issues/8243#issuecomment-631651715 This is addressed by #7232. We were able to apply the above patch and observe it resolving this issue for us.

[jira] [Commented] (AIRFLOW-6983) Disabled connection pool in CLI might be harmful

2020-05-20 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112507#comment-17112507 ] ASF GitHub Bot commented on AIRFLOW-6983: - ashb commented on pull request #7622: URL:

[jira] [Commented] (AIRFLOW-6983) Disabled connection pool in CLI might be harmful

2020-05-20 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112506#comment-17112506 ] ASF GitHub Bot commented on AIRFLOW-6983: - ashb commented on a change in pull request #7622:

[GitHub] [airflow] ashb commented on pull request #7622: [AIRFLOW-6983] Use SingletonThreadPool for database communication

2020-05-20 Thread GitBox
ashb commented on pull request #7622: URL: https://github.com/apache/airflow/pull/7622#issuecomment-631641575 @Khrol ci having some general issues today , should be fixed in an hour hopefully This is an automated message

[GitHub] [airflow] ashb commented on a change in pull request #7622: [AIRFLOW-6983] Use SingletonThreadPool for database communication

2020-05-20 Thread GitBox
ashb commented on a change in pull request #7622: URL: https://github.com/apache/airflow/pull/7622#discussion_r428213605 ## File path: airflow/settings.py ## @@ -148,18 +148,6 @@ def configure_orm(disable_connection_pool=False): # 0 means no limit, which could lead to

[GitHub] [airflow] feluelle commented on a change in pull request #8895: Add Delete/Create S3 bucket operators

2020-05-20 Thread GitBox
feluelle commented on a change in pull request #8895: URL: https://github.com/apache/airflow/pull/8895#discussion_r428206674 ## File path: airflow/providers/amazon/aws/operators/s3_bucket.py ## @@ -0,0 +1,119 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one

[jira] [Commented] (AIRFLOW-6569) Broken sentry integration

2020-05-20 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112497#comment-17112497 ] ASF GitHub Bot commented on AIRFLOW-6569: - mikeclarke commented on pull request #7232: URL:

[GitHub] [airflow] feluelle commented on a change in pull request #8895: Add Delete/Create S3 bucket operators

2020-05-20 Thread GitBox
feluelle commented on a change in pull request #8895: URL: https://github.com/apache/airflow/pull/8895#discussion_r428206674 ## File path: airflow/providers/amazon/aws/operators/s3_bucket.py ## @@ -0,0 +1,119 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [airflow] feluelle commented on a change in pull request #8895: Add Delete/Create S3 bucket operators

2020-05-20 Thread GitBox
feluelle commented on a change in pull request #8895: URL: https://github.com/apache/airflow/pull/8895#discussion_r428206674 ## File path: airflow/providers/amazon/aws/operators/s3_bucket.py ## @@ -0,0 +1,119 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one

[GitHub] [airflow] mikeclarke commented on pull request #7232: [AIRFLOW-6569] Flush pending Sentry exceptions before exiting forked process

2020-05-20 Thread GitBox
mikeclarke commented on pull request #7232: URL: https://github.com/apache/airflow/pull/7232#issuecomment-631635519 Rebased master just now - last time the tests were failing for unrelated reasons, but we'll see. This is an

[GitHub] [airflow] boring-cyborg[bot] commented on issue #8930: Jenkins connection edit is not working on Web Server

2020-05-20 Thread GitBox
boring-cyborg[bot] commented on issue #8930: URL: https://github.com/apache/airflow/issues/8930#issuecomment-631632733 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] ptamorri opened a new issue #8930: Jenkins connection edit is not working on Web Server

2020-05-20 Thread GitBox
ptamorri opened a new issue #8930: URL: https://github.com/apache/airflow/issues/8930 Hi to all, i'm using Airflow version 1.10.10 (the actual latest) and get error page everytime i try to modify a Jenkins connection. If i delete it and recreate it, it works. I'm using

[GitHub] [airflow] chrismclennon commented on pull request #8914: Add "Next scheduled run" to Tree View

2020-05-20 Thread GitBox
chrismclennon commented on pull request #8914: URL: https://github.com/apache/airflow/pull/8914#issuecomment-631629584 Thanks @ash! This is an automated message from the Apache Git Service. To respond to the message, please

  1   2   3   4   >