[GitHub] [airflow] mik-laj edited a comment on pull request #7197: [AIRFLOW-6586] Improvements to gcs sensor

2020-05-03 Thread GitBox
mik-laj edited a comment on pull request #7197: URL: https://github.com/apache/airflow/pull/7197#issuecomment-623267591 This decorator should be in the same module as BaseSensorOperato. If you want to write some documentation, you can also add information about it to the guide on writing

[jira] [Commented] (AIRFLOW-6586) GCSUploadSessionCompleteSensor breaks in reschedule mode.

2020-05-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098692#comment-17098692 ] ASF GitHub Bot commented on AIRFLOW-6586: - mik-laj commented on pull request #7197: URL:

[GitHub] [airflow] mik-laj commented on pull request #7197: [AIRFLOW-6586] Improvements to gcs sensor

2020-05-03 Thread GitBox
mik-laj commented on pull request #7197: URL: https://github.com/apache/airflow/pull/7197#issuecomment-623267591 This decorator should be in the same module as BaseSensorOperator If you want to write some documentation, you can add information about it to the guide on writing

[jira] [Commented] (AIRFLOW-6586) GCSUploadSessionCompleteSensor breaks in reschedule mode.

2020-05-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098693#comment-17098693 ] ASF GitHub Bot commented on AIRFLOW-6586: - mik-laj edited a comment on pull request #7197: URL:

[GitHub] [airflow] mik-laj commented on a change in pull request #7197: [AIRFLOW-6586] Improvements to gcs sensor

2020-05-03 Thread GitBox
mik-laj commented on a change in pull request #7197: URL: https://github.com/apache/airflow/pull/7197#discussion_r419218559 ## File path: airflow/sensors/decorators/poke_mode_only.py ## @@ -0,0 +1,80 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or

[jira] [Commented] (AIRFLOW-6586) GCSUploadSessionCompleteSensor breaks in reschedule mode.

2020-05-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098689#comment-17098689 ] ASF GitHub Bot commented on AIRFLOW-6586: - mik-laj commented on a change in pull request #7197:

[GitHub] [airflow] mik-laj commented on a change in pull request #7197: [AIRFLOW-6586] Improvements to gcs sensor

2020-05-03 Thread GitBox
mik-laj commented on a change in pull request #7197: URL: https://github.com/apache/airflow/pull/7197#discussion_r419220670 ## File path: airflow/sensors/decorators/poke_mode_only.py ## @@ -0,0 +1,80 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or

[jira] [Commented] (AIRFLOW-6586) GCSUploadSessionCompleteSensor breaks in reschedule mode.

2020-05-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098688#comment-17098688 ] ASF GitHub Bot commented on AIRFLOW-6586: - mik-laj commented on a change in pull request #7197:

[jira] [Commented] (AIRFLOW-6586) GCSUploadSessionCompleteSensor breaks in reschedule mode.

2020-05-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098684#comment-17098684 ] ASF GitHub Bot commented on AIRFLOW-6586: - mik-laj commented on a change in pull request #7197:

[GitHub] [airflow] mik-laj commented on a change in pull request #7197: [AIRFLOW-6586] Improvements to gcs sensor

2020-05-03 Thread GitBox
mik-laj commented on a change in pull request #7197: URL: https://github.com/apache/airflow/pull/7197#discussion_r419218559 ## File path: airflow/sensors/decorators/poke_mode_only.py ## @@ -0,0 +1,80 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or

[GitHub] [airflow] mik-laj commented on a change in pull request #8535: [Airflow-8439] Refactor test_variable_command.py

2020-05-03 Thread GitBox
mik-laj commented on a change in pull request #8535: URL: https://github.com/apache/airflow/pull/8535#discussion_r419217277 ## File path: tests/cli/commands/test_variable_command.py ## @@ -31,58 +34,38 @@ def setUpClass(cls): cls.dagbag =

[GitHub] [airflow] seanxwzhang commented on a change in pull request #8535: [Airflow-8439] Refactor test_variable_command.py

2020-05-03 Thread GitBox
seanxwzhang commented on a change in pull request #8535: URL: https://github.com/apache/airflow/pull/8535#discussion_r419210541 ## File path: tests/cli/commands/test_variable_command.py ## @@ -31,58 +34,38 @@ def setUpClass(cls): cls.dagbag =

[jira] [Commented] (AIRFLOW-4761) Airflow Task Clear function throws error

2020-05-03 Thread Kamil Choudhury (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-4761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098649#comment-17098649 ] Kamil Choudhury commented on AIRFLOW-4761: -- I'm still running into this on 1.10.10/python

[GitHub] [airflow] RosterIn commented on issue #8696: Skip task itself instead of all downstream tasks

2020-05-03 Thread GitBox
RosterIn commented on issue #8696: URL: https://github.com/apache/airflow/issues/8696#issuecomment-623248308 seems related to https://github.com/apache/airflow/issues/7858 This is an automated message from the Apache Git

[GitHub] [airflow] mik-laj commented on a change in pull request #8651: [AIP-31] [AIRFLOW-8058] Retrieve current execution context without altering function's signature

2020-05-03 Thread GitBox
mik-laj commented on a change in pull request #8651: URL: https://github.com/apache/airflow/pull/8651#discussion_r419128640 ## File path: tests/task/context/test_current_context.py ## @@ -0,0 +1,100 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more

[GitHub] [airflow] mik-laj commented on pull request #8694: Add airflow config command (v1-10-test)

2020-05-03 Thread GitBox
mik-laj commented on pull request #8694: URL: https://github.com/apache/airflow/pull/8694#issuecomment-623248081 Checks are green :-) This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [airflow] RosterIn commented on pull request #8668: Bump supported mysqlclient to <1.5

2020-05-03 Thread GitBox
RosterIn commented on pull request #8668: URL: https://github.com/apache/airflow/pull/8668#issuecomment-623247580 you have erros: ``` ImportError while importing test module '/opt/airflow/tests/providers/google/cloud/operators/test_mysql_to_gcs.py'. Hint: make sure your test

[GitHub] [airflow] RosterIn commented on pull request #8154: Delete cast(DR.execution_date, DateTime)

2020-05-03 Thread GitBox
RosterIn commented on pull request #8154: URL: https://github.com/apache/airflow/pull/8154#issuecomment-623245883 > The cast was added by #1871, but it prevents sqlalchemy from automatically converting execution_date to UTC and fails

[GitHub] [airflow] RosterIn commented on pull request #8226: Update ldap_auth.py

2020-05-03 Thread GitBox
RosterIn commented on pull request #8226: URL: https://github.com/apache/airflow/pull/8226#issuecomment-623243647 @ShanGor why this PR is against v1.10 branch? This is an automated message from the Apache Git Service. To

[GitHub] [airflow] gdevanla commented on pull request #8699: Scheduler just checks for task instances in 'running' state in execution. #8691

2020-05-03 Thread GitBox
gdevanla commented on pull request #8699: URL: https://github.com/apache/airflow/pull/8699#issuecomment-623242313 I have added this a proof-of-concept change. Any reviews regarding this approach is appreciated. I will add tests around this scenario if this is the change we want to make.

[GitHub] [airflow] gdevanla opened a new pull request #8699: Scheduler just checks for task instances in 'running' state in execution. #8691

2020-05-03 Thread GitBox
gdevanla opened a new pull request #8699: URL: https://github.com/apache/airflow/pull/8699 The 'scheduler' needs to only check if the executor has started the task by checked the 'running' dictionary of the executor instance. The entry in the `Executor.queued_tasks` will be over-written

[GitHub] [airflow] lokeshlal commented on pull request #8509: [AIRFLOW-4568]The ExternalTaskSensor should be configurable to raise an Airflow Exception in case the poked external task reaches a disall

2020-05-03 Thread GitBox
lokeshlal commented on pull request #8509: URL: https://github.com/apache/airflow/pull/8509#issuecomment-623240562 Hello @potiuk Could you please help in reviewing and merging this PR. Thanks. This is an automated message

[jira] [Commented] (AIRFLOW-4568) The ExternalTaskSensor should be configurable to raise an Airflow Exception in case the poked external task reaches a disallowed state, such as f.i. failed

2020-05-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-4568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098640#comment-17098640 ] ASF GitHub Bot commented on AIRFLOW-4568: - lokeshlal commented on pull request #8509: URL:

[GitHub] [airflow] mik-laj opened a new pull request #8698: Fix failed master - Invalid output in test_variable assertion

2020-05-03 Thread GitBox
mik-laj opened a new pull request #8698: URL: https://github.com/apache/airflow/pull/8698 --- Make sure to mark the boxes below before creating PR: [x] - [X] Description above provides context of the change - [X] Unit tests coverage for changes (not needed for documentation

[GitHub] [airflow] jaketf commented on pull request #7197: [AIRFLOW-6586] Improvements to gcs sensor

2020-05-03 Thread GitBox
jaketf commented on pull request #7197: URL: https://github.com/apache/airflow/pull/7197#issuecomment-623235491 @turbaszek this is the first decorator in this dir. How do you think we should document this? In the same place the docs ci check is complaining about ?

[jira] [Commented] (AIRFLOW-6586) GCSUploadSessionCompleteSensor breaks in reschedule mode.

2020-05-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098635#comment-17098635 ] ASF GitHub Bot commented on AIRFLOW-6586: - jaketf commented on pull request #7197: URL:

[GitHub] [airflow] maganaluis commented on a change in pull request #8256: updated _write_args on PythonVirtualenvOperator

2020-05-03 Thread GitBox
maganaluis commented on a change in pull request #8256: URL: https://github.com/apache/airflow/pull/8256#discussion_r419189015 ## File path: airflow/operators/python_operator.py ## @@ -330,13 +330,28 @@ def _write_string_args(self, filename): def _write_args(self,

[GitHub] [airflow] resdevd edited a comment on pull request #8621: Docker Compose CeleryExecutor example

2020-05-03 Thread GitBox
resdevd edited a comment on pull request #8621: URL: https://github.com/apache/airflow/pull/8621#issuecomment-623221335 entrypoint.sh is maintained in dockerfile itself. This PR is docker-compose setup using Airflow official Docker Image. If you explore the code, you can see one time

[GitHub] [airflow] maganaluis commented on a change in pull request #8256: updated _write_args on PythonVirtualenvOperator

2020-05-03 Thread GitBox
maganaluis commented on a change in pull request #8256: URL: https://github.com/apache/airflow/pull/8256#discussion_r419189015 ## File path: airflow/operators/python_operator.py ## @@ -330,13 +330,28 @@ def _write_string_args(self, filename): def _write_args(self,

[GitHub] [airflow] resdevd edited a comment on pull request #8621: Docker Compose CeleryExecutor example

2020-05-03 Thread GitBox
resdevd edited a comment on pull request #8621: URL: https://github.com/apache/airflow/pull/8621#issuecomment-623221335 entrypoint.sh is maintained in dockerfile itself. This PR is docker-compose setup using Airflow official Docker Image. If you explore the code, you can see one time

[GitHub] [airflow] resdevd commented on pull request #8621: Docker Compose CeleryExecutor example

2020-05-03 Thread GitBox
resdevd commented on pull request #8621: URL: https://github.com/apache/airflow/pull/8621#issuecomment-623221335 entrypoint.sh is maintained in dockerfile itself. This PR is docker-compose setup using Airflow official Docker Image. If you explore the code, you can see one time setup of

[GitHub] [airflow] mik-laj opened a new pull request #8697: Changea provider:GCP to provider:Google

2020-05-03 Thread GitBox
mik-laj opened a new pull request #8697: URL: https://github.com/apache/airflow/pull/8697 It seems to me that it has a lot in common. I am also sad because the changes in the base hook have no label. We need to change name of the label in Github before merging this change.

[jira] [Commented] (AIRFLOW-7008) Add perf kit with common used decorators/contexts

2020-05-03 Thread ASF subversion and git services (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098617#comment-17098617 ] ASF subversion and git services commented on AIRFLOW-7008: -- Commit

[jira] [Commented] (AIRFLOW-7008) Add perf kit with common used decorators/contexts

2020-05-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098616#comment-17098616 ] ASF GitHub Bot commented on AIRFLOW-7008: - mik-laj commented on a change in pull request #7650:

[GitHub] [airflow] mik-laj commented on a change in pull request #7650: [AIRFLOW-7008] Add perf kit with common used decorators/contexts

2020-05-03 Thread GitBox
mik-laj commented on a change in pull request #7650: URL: https://github.com/apache/airflow/pull/7650#discussion_r419183415 ## File path: scripts/perf/perf_kit/python.py ## @@ -0,0 +1,108 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

[GitHub] [airflow] mik-laj commented on issue #8659: Setting num_retries for a google-cloud-platform using env variables breaks google tasks

2020-05-03 Thread GitBox
mik-laj commented on issue #8659: URL: https://github.com/apache/airflow/issues/8659#issuecomment-623213202 @michalslowikowski00 Can you help with it? This is an automated message from the Apache Git Service. To respond to

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #8556: Add system test for gcs_to_bigquery

2020-05-03 Thread GitBox
boring-cyborg[bot] commented on pull request #8556: URL: https://github.com/apache/airflow/pull/8556#issuecomment-623212977 Awesome work, congrats on your first merged pull request! This is an automated message from the

[GitHub] [airflow] mik-laj commented on a change in pull request #6277: [AIRFLOW-2971] Add health check CLI for scheduler

2020-05-03 Thread GitBox
mik-laj commented on a change in pull request #6277: URL: https://github.com/apache/airflow/pull/6277#discussion_r419176878 ## File path: tests/cli/test_cli.py ## @@ -488,3 +490,26 @@ def test_run_naive_taskinstance(self, mock_local_job): pickle_id=None,

[jira] [Commented] (AIRFLOW-2971) Health check command for scheduler

2020-05-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098613#comment-17098613 ] ASF GitHub Bot commented on AIRFLOW-2971: - mik-laj commented on a change in pull request #6277:

[GitHub] [airflow] mik-laj commented on pull request #8653: [AIRFLOW-8645] BQ: integer range partitioning support for BQ tables

2020-05-03 Thread GitBox
mik-laj commented on pull request #8653: URL: https://github.com/apache/airflow/pull/8653#issuecomment-623208822 Hi. Integration with BigQuery is undergoing extensive refactor by @turbaszek . Can you wait until we finish this work? It is possible that then this change will not be

[GitHub] [airflow] mik-laj commented on issue #8647: Import fails when dags subfolder and plugins subfolder have the same name

2020-05-03 Thread GitBox
mik-laj commented on issue #8647: URL: https://github.com/apache/airflow/issues/8647#issuecomment-623208206 This is the expected behavior. Airflow adds **an additional 3 directories** to the variable `sys.path`. * ``conf.get('core', 'dags_folder')`` * ``conf.get('core',

[GitHub] [airflow] boring-cyborg[bot] commented on issue #8696: Skip task itself instead of all downstream tasks

2020-05-03 Thread GitBox
boring-cyborg[bot] commented on issue #8696: URL: https://github.com/apache/airflow/issues/8696#issuecomment-623207419 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] yuqian90 opened a new issue #8696: Skip task itself instead of all downstream tasks

2020-05-03 Thread GitBox
yuqian90 opened a new issue #8696: URL: https://github.com/apache/airflow/issues/8696 **Apache Airflow version**: 1.10.10 **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: - **Cloud provider or hardware configuration**:

[GitHub] [airflow] mik-laj commented on pull request #8679: Fix displaying Executor Class Name in "Base Job" table

2020-05-03 Thread GitBox
mik-laj commented on pull request #8679: URL: https://github.com/apache/airflow/pull/8679#issuecomment-623205033 Why does every job need an executor? I think LocalTaskJob doesn't need it. This is an automated message from

[GitHub] [airflow] mik-laj commented on a change in pull request #6277: [AIRFLOW-2971] Add health check CLI for scheduler

2020-05-03 Thread GitBox
mik-laj commented on a change in pull request #6277: URL: https://github.com/apache/airflow/pull/6277#discussion_r419176878 ## File path: tests/cli/test_cli.py ## @@ -488,3 +490,26 @@ def test_run_naive_taskinstance(self, mock_local_job): pickle_id=None,

[jira] [Commented] (AIRFLOW-2971) Health check command for scheduler

2020-05-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-2971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098601#comment-17098601 ] ASF GitHub Bot commented on AIRFLOW-2971: - mik-laj commented on a change in pull request #6277:

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #8640: Support k8s auth method in vault secrets provider

2020-05-03 Thread GitBox
boring-cyborg[bot] commented on pull request #8640: URL: https://github.com/apache/airflow/pull/8640#issuecomment-623198724 Awesome work, congrats on your first merged pull request! This is an automated message from the

[GitHub] [airflow] kaxil commented on pull request #8505: [POC] Add more precise tests (tests for jinja params) for AirflowVersion

2020-05-03 Thread GitBox
kaxil commented on pull request #8505: URL: https://github.com/apache/airflow/pull/8505#issuecomment-623195226 For future reference: https://flask.palletsprojects.com/en/1.1.x/signals/#subscribing-to-signals This is an

[GitHub] [airflow] jsmodic edited a comment on issue #8480: Celery autoscaling overrides normal worker_concurrency setting

2020-05-03 Thread GitBox
jsmodic edited a comment on issue #8480: URL: https://github.com/apache/airflow/issues/8480#issuecomment-623192154 As far as I tested it should be enough to remove 'autoscale' from the dictionary to be unpacked into Celery if autoscaling isn't configured in the airflow.cfg. But I haven't

[GitHub] [airflow] jsmodic commented on issue #8480: Celery autoscaling overrides normal worker_concurrency setting

2020-05-03 Thread GitBox
jsmodic commented on issue #8480: URL: https://github.com/apache/airflow/issues/8480#issuecomment-623192154 I can't speak to how Celery's various versions work, but as far as I tested it should be enough to remove 'autoscale' from the dictionary to be unpacked into Celery if autoscaling

[GitHub] [airflow] mik-laj commented on pull request #8505: [POC] Add more precise tests (tests for jinja params) for AirflowVersion

2020-05-03 Thread GitBox
mik-laj commented on pull request #8505: URL: https://github.com/apache/airflow/pull/8505#issuecomment-623191262 @potiuk @ashb @kaxil @turbaszek I have updated PR. Now I use flask.signals instead of mock. This is an

[GitHub] [airflow] mik-laj commented on a change in pull request #8505: [POC] Add more precise tests (tests for jinja params) for AirflowVersion

2020-05-03 Thread GitBox
mik-laj commented on a change in pull request #8505: URL: https://github.com/apache/airflow/pull/8505#discussion_r419167999 ## File path: setup.py ## @@ -427,6 +427,7 @@ def write_version(filename: str = os.path.join(*[my_dir, "airflow", "git_version

[GitHub] [airflow] mik-laj commented on pull request #8694: Add airflow config command (v1-10-test)

2020-05-03 Thread GitBox
mik-laj commented on pull request #8694: URL: https://github.com/apache/airflow/pull/8694#issuecomment-623188397 I'm working on it. This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [airflow] nadflinn commented on issue #8480: Celery autoscaling overrides normal worker_concurrency setting

2020-05-03 Thread GitBox
nadflinn commented on issue #8480: URL: https://github.com/apache/airflow/issues/8480#issuecomment-623185999 The autoscale config is commented out since 1.10.10 which is a change from 1.10.9 where it isn't and I think this accounts for the difference in the behavior that people are

[GitHub] [airflow] gpongracz commented on pull request #8621: Docker Compose CeleryExecutor example

2020-05-03 Thread GitBox
gpongracz commented on pull request #8621: URL: https://github.com/apache/airflow/pull/8621#issuecomment-623185001 Looks like the entrypoint.sh is not doing an initdb or upgradedb prior to the command like is found in puckel

[GitHub] [airflow] nadflinn opened a new pull request #8695: Celery worker prefetch multiplier configurable

2020-05-03 Thread GitBox
nadflinn opened a new pull request #8695: URL: https://github.com/apache/airflow/pull/8695 There is some debate about whether Celery autoscale actually works, as discussed in the comments to this PR: https://github.com/apache/airflow/pull/3989#issuecomment-535882666 This is also

[GitHub] [airflow] kaxil commented on pull request #8694: Add airflow config command (v1-10-test)

2020-05-03 Thread GitBox
kaxil commented on pull request #8694: URL: https://github.com/apache/airflow/pull/8694#issuecomment-623184476 Thanks for backporting the fixes but looks like the tests are failing. This is an automated message from the

[GitHub] [airflow] habibdhif commented on issue #8605: Add Production-ready docker compose for the production image

2020-05-03 Thread GitBox
habibdhif commented on issue #8605: URL: https://github.com/apache/airflow/issues/8605#issuecomment-623182960 Here is another example of a Docker Compose that I've been working on. The Compose defines multiple services to run Airflow. There is an init service which is an ephemeral

[GitHub] [airflow] mik-laj edited a comment on pull request #8305: Guide for Apache Spark operators

2020-05-03 Thread GitBox
mik-laj edited a comment on pull request #8305: URL: https://github.com/apache/airflow/pull/8305#issuecomment-623137703 One more thing. Can you add this tutorial to `operators-and-hooks-ref.rst`? https://github.com/apache/airflow/pull/8690 I am working to automatically check that the

[jira] [Commented] (AIRFLOW-6440) AWS Fargate Executor (AIP-29) (WIP)

2020-05-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-6440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098551#comment-17098551 ] ASF GitHub Bot commented on AIRFLOW-6440: - stale[bot] commented on pull request #7030: URL:

[GitHub] [airflow] stale[bot] commented on pull request #7030: [AIRFLOW-6440][AIP-29] Add AWS Fargate Executor

2020-05-03 Thread GitBox
stale[bot] commented on pull request #7030: URL: https://github.com/apache/airflow/pull/7030#issuecomment-623172002 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jira] [Commented] (AIRFLOW-7065) Give StreamLogHandler ability to tee to log and console

2020-05-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098545#comment-17098545 ] ASF GitHub Bot commented on AIRFLOW-7065: - c-wilson commented on a change in pull request

[GitHub] [airflow] c-wilson commented on a change in pull request #7740: [AIRFLOW-7065] Add optional propagation of task std streams to console

2020-05-03 Thread GitBox
c-wilson commented on a change in pull request #7740: URL: https://github.com/apache/airflow/pull/7740#discussion_r419148387 ## File path: airflow/utils/log/logging_mixin.py ## @@ -118,6 +129,53 @@ def isatty(self): """ return False +def

[GitHub] [airflow] habibdhif commented on pull request #8689: [AIRFLOW-8605] Added docker compose

2020-05-03 Thread GitBox
habibdhif commented on pull request #8689: URL: https://github.com/apache/airflow/pull/8689#issuecomment-623166794 > Note there's another PR for a docker-compose file: #8621 Thanks for notifying me, closing the PR and moving this example under the #8605 issue.

[GitHub] [airflow] BasPH commented on pull request #8689: [AIRFLOW-8605] Added docker compose

2020-05-03 Thread GitBox
BasPH commented on pull request #8689: URL: https://github.com/apache/airflow/pull/8689#issuecomment-623166409 Note there's another PR for a docker-compose file: https://github.com/apache/airflow/pull/8621 This is an

[GitHub] [airflow] c-wilson commented on pull request #7740: [AIRFLOW-7065] Add optional propagation of task std streams to console

2020-05-03 Thread GitBox
c-wilson commented on pull request #7740: URL: https://github.com/apache/airflow/pull/7740#issuecomment-623164289 Hi @ashb I made some trivial cleanup changes to the ESTaskHandler but in reality I cannot find a better way to implement its functionality with the current (slightly deep)

[jira] [Commented] (AIRFLOW-7065) Give StreamLogHandler ability to tee to log and console

2020-05-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098542#comment-17098542 ] ASF GitHub Bot commented on AIRFLOW-7065: - c-wilson commented on pull request #7740: URL:

[GitHub] [airflow] mik-laj commented on a change in pull request #7740: [AIRFLOW-7065] Add optional propagation of task std streams to console

2020-05-03 Thread GitBox
mik-laj commented on a change in pull request #7740: URL: https://github.com/apache/airflow/pull/7740#discussion_r419145400 ## File path: airflow/utils/log/logging_mixin.py ## @@ -118,6 +129,53 @@ def isatty(self): """ return False +def

[jira] [Commented] (AIRFLOW-7065) Give StreamLogHandler ability to tee to log and console

2020-05-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098539#comment-17098539 ] ASF GitHub Bot commented on AIRFLOW-7065: - mik-laj commented on a change in pull request #7740:

[jira] [Commented] (AIRFLOW-7065) Give StreamLogHandler ability to tee to log and console

2020-05-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098538#comment-17098538 ] ASF GitHub Bot commented on AIRFLOW-7065: - mik-laj commented on a change in pull request #7740:

[GitHub] [airflow] mik-laj commented on a change in pull request #7740: [AIRFLOW-7065] Add optional propagation of task std streams to console

2020-05-03 Thread GitBox
mik-laj commented on a change in pull request #7740: URL: https://github.com/apache/airflow/pull/7740#discussion_r419145161 ## File path: airflow/utils/log/logging_mixin.py ## @@ -118,6 +129,53 @@ def isatty(self): """ return False +def

[GitHub] [airflow] XD-DENG commented on pull request #8692: Sort connection type list in add/edit page alphabetically

2020-05-03 Thread GitBox
XD-DENG commented on pull request #8692: URL: https://github.com/apache/airflow/pull/8692#issuecomment-623156345 Thanks @zhongjiajie and @mik-laj This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] mustafagok commented on pull request #8581: Add google_api_to_s3_transfer example dags and system tests

2020-05-03 Thread GitBox
mustafagok commented on pull request #8581: URL: https://github.com/apache/airflow/pull/8581#issuecomment-623156223 I see that you have much bigger problems, I would like to suggest a small change, about naming "aws_system_helpers.py" as "amazon_system_helpers.py" since it contains

[GitHub] [airflow] mik-laj opened a new pull request #8694: Add airfloww config command (v1-10-test)

2020-05-03 Thread GitBox
mik-laj opened a new pull request #8694: URL: https://github.com/apache/airflow/pull/8694 This PR is based on the following PRs: https://github.com/apache/airflow/pull/6840 https://github.com/apache/airflow/pull/7117 https://github.com/apache/airflow/pull/8404

[GitHub] [airflow] mustafagok commented on a change in pull request #8618: Add AWS EMR System tests

2020-05-03 Thread GitBox
mustafagok commented on a change in pull request #8618: URL: https://github.com/apache/airflow/pull/8618#discussion_r419136967 ## File path: tests/providers/amazon/aws/operators/test_emr_system.py ## @@ -0,0 +1,47 @@ +# Licensed to the Apache Software Foundation (ASF) under

[GitHub] [airflow] mustafagok commented on a change in pull request #8618: Add AWS EMR System tests

2020-05-03 Thread GitBox
mustafagok commented on a change in pull request #8618: URL: https://github.com/apache/airflow/pull/8618#discussion_r419136343 ## File path: tests/providers/amazon/aws/operators/test_emr_system.py ## @@ -0,0 +1,47 @@ +# Licensed to the Apache Software Foundation (ASF) under

[GitHub] [airflow] mik-laj commented on pull request #6075: [AIRFLOW-5266] Allow aws_athena_hook to get all query results

2020-05-03 Thread GitBox
mik-laj commented on pull request #6075: URL: https://github.com/apache/airflow/pull/6075#issuecomment-623149528 @lindsable Do you need any help? If you have applied all the comments of the reviewer, then it is worth asking to review again using a comment or PM on Slack. The

[jira] [Commented] (AIRFLOW-5266) AWS Athena Hook only returns first 1000 results

2020-05-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-5266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098514#comment-17098514 ] ASF GitHub Bot commented on AIRFLOW-5266: - mik-laj commented on pull request #6075: URL:

[GitHub] [airflow] mik-laj commented on pull request #7777: [AIRFLOW-Date_range compatible with schedule_interval]

2020-05-03 Thread GitBox
mik-laj commented on pull request #: URL: https://github.com/apache/airflow/pull/#issuecomment-623148339 @Rcharriol Have you finished working on this change? I see that Travis is sad, but in the meantime we migrated to Github Action. Can you do rebase? If you have applied

[GitHub] [airflow] mik-laj edited a comment on pull request #7777: [AIRFLOW-Date_range compatible with schedule_interval]

2020-05-03 Thread GitBox
mik-laj edited a comment on pull request #: URL: https://github.com/apache/airflow/pull/#issuecomment-623148339 @Rcharriol Have you finished working on this change? I see that Travis is sad, but in the meantime we migrated to Github Action. Can you do rebase? If you have

[jira] [Commented] (AIRFLOW-7068) Create EC2 Hook, Operator and Sensor

2020-05-03 Thread ASF GitHub Bot (Jira)
[ https://issues.apache.org/jira/browse/AIRFLOW-7068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098508#comment-17098508 ] ASF GitHub Bot commented on AIRFLOW-7068: - mik-laj commented on pull request #7731: URL:

[GitHub] [airflow] mik-laj commented on pull request #7731: [AIRFLOW-7068] Create EC2 Hook, Operator and Sensor

2020-05-03 Thread GitBox
mik-laj commented on pull request #7731: URL: https://github.com/apache/airflow/pull/7731#issuecomment-623147774 @mustafagok What is the status of this PR? Do you need any help? If you have applied all the comments of the reviewer, then it is worth asking to review again using a comment

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #8535: [Airflow-8439] Refactor test_variable_command.py

2020-05-03 Thread GitBox
boring-cyborg[bot] commented on pull request #8535: URL: https://github.com/apache/airflow/pull/8535#issuecomment-623146753 Awesome work, congrats on your first merged pull request! This is an automated message from the

[GitHub] [airflow] mik-laj opened a new pull request #8693: Carefully parse warning messages when building documentation

2020-05-03 Thread GitBox
mik-laj opened a new pull request #8693: URL: https://github.com/apache/airflow/pull/8693 --- Make sure to mark the boxes below before creating PR: [x] - [X] Description above provides context of the change - [X] Unit tests coverage for changes (not needed for documentation

[GitHub] [airflow] XD-DENG commented on pull request #8685: Fix Spark Conn Add/Edit Pages

2020-05-03 Thread GitBox
XD-DENG commented on pull request #8685: URL: https://github.com/apache/airflow/pull/8685#issuecomment-623144773 Hi @zhongjiajie , I have split the PR into two. The change to order connection types alphabetically is in PR https://github.com/apache/airflow/pull/8692 The PR

[GitHub] [airflow] XD-DENG opened a new pull request #8692: Sort connection type list in add/edit page alphabetically

2020-05-03 Thread GitBox
XD-DENG opened a new pull request #8692: URL: https://github.com/apache/airflow/pull/8692 Currently the connection type list in the UI is sorted in the order of original `Connection._types`, which may be a bit inconvenient for users. It would be better if it can be sorted

[GitHub] [airflow] XD-DENG commented on pull request #8685: Fix Spark Conn Add/Edit Pages + Sort Connection Type Options

2020-05-03 Thread GitBox
XD-DENG commented on pull request #8685: URL: https://github.com/apache/airflow/pull/8685#issuecomment-623142738 > Code looking good, but should we separate PR? Seem sorting and adding miss type unrelated. @zhongjiajie two changes are both small so appended the 2nd change here. But

[GitHub] [airflow] mik-laj commented on a change in pull request #8651: [AIP-31] [AIRFLOW-8058] Retrieve current execution context without altering function's signature

2020-05-03 Thread GitBox
mik-laj commented on a change in pull request #8651: URL: https://github.com/apache/airflow/pull/8651#discussion_r419129507 ## File path: airflow/config_templates/config.yml ## @@ -366,6 +366,17 @@ type: string example: "path.to.CustomXCom" default:

[GitHub] [airflow] mik-laj commented on a change in pull request #8651: [AIP-31] [AIRFLOW-8058] Retrieve current execution context without altering function's signature

2020-05-03 Thread GitBox
mik-laj commented on a change in pull request #8651: URL: https://github.com/apache/airflow/pull/8651#discussion_r419128640 ## File path: tests/task/context/test_current_context.py ## @@ -0,0 +1,100 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more

[GitHub] [airflow] boring-cyborg[bot] commented on issue #8691: Task Instance will not be queued if `celery_executor` does not drop queue tasks on error.

2020-05-03 Thread GitBox
boring-cyborg[bot] commented on issue #8691: URL: https://github.com/apache/airflow/issues/8691#issuecomment-623140181 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated

[GitHub] [airflow] gdevanla opened a new issue #8691: Task Instance will not be queued if `celery_executor` does not drop queue tasks on error.

2020-05-03 Thread GitBox
gdevanla opened a new issue #8691: URL: https://github.com/apache/airflow/issues/8691 **Apache Airflow version**: 1.10.3 **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: Python 3.7.4 - **Cloud provider or hardware

[GitHub] [airflow] mik-laj commented on a change in pull request #8651: [AIP-31] [AIRFLOW-8058] Retrieve current execution context without altering function's signature

2020-05-03 Thread GitBox
mik-laj commented on a change in pull request #8651: URL: https://github.com/apache/airflow/pull/8651#discussion_r419128190 ## File path: airflow/models/taskinstance.py ## @@ -1110,6 +1116,27 @@ def signal_handler(signum, frame): session.merge(self)

[GitHub] [airflow] mik-laj commented on a change in pull request #8651: [AIP-31] [AIRFLOW-8058] Retrieve current execution context without altering function's signature

2020-05-03 Thread GitBox
mik-laj commented on a change in pull request #8651: URL: https://github.com/apache/airflow/pull/8651#discussion_r419128100 ## File path: airflow/models/taskinstance.py ## @@ -1110,6 +1116,27 @@ def signal_handler(signum, frame): session.merge(self)

[GitHub] [airflow] mik-laj commented on pull request #8305: Guide for Apache Spark operators

2020-05-03 Thread GitBox
mik-laj commented on pull request #8305: URL: https://github.com/apache/airflow/pull/8305#issuecomment-623137703 One more thing. Can you add this tutorial to `operators-and-hooks-ref.rst`? https://github.com/apache/airflow/pull/8690/files I am working to automatically check that the

[GitHub] [airflow] mik-laj commented on a change in pull request #8305: Guide for Apache Spark operators

2020-05-03 Thread GitBox
mik-laj commented on a change in pull request #8305: URL: https://github.com/apache/airflow/pull/8305#discussion_r419126976 ## File path: docs/howto/operator/apache/spark.rst ## @@ -0,0 +1,102 @@ + .. Licensed to the Apache Software Foundation (ASF) under one +or more

[GitHub] [airflow] mik-laj opened a new pull request #8690: Check consistency between reference file and howto directory

2020-05-03 Thread GitBox
mik-laj opened a new pull request #8690: URL: https://github.com/apache/airflow/pull/8690 Depends on: https://github.com/apache/airflow/pull/8623 --- Make sure to mark the boxes below before creating PR: [x] - [X] Description above provides context of the change - [X] Unit

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #8670: Include all imports in `import_module` block

2020-05-03 Thread GitBox
boring-cyborg[bot] commented on pull request #8670: URL: https://github.com/apache/airflow/pull/8670#issuecomment-623135851 Awesome work, congrats on your first merged pull request! This is an automated message from the

[GitHub] [airflow] mik-laj commented on a change in pull request #8689: [AIRFLOW-8605] Added docker compose

2020-05-03 Thread GitBox
mik-laj commented on a change in pull request #8689: URL: https://github.com/apache/airflow/pull/8689#discussion_r419126176 ## File path: IMAGES.rst ## @@ -416,3 +416,22 @@ signals). This entrypoint works as follows: * If first argument is equal to "bash" - you are dropped

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #8677: Remove _get_pretty_exception_message in PrestoHook

2020-05-03 Thread GitBox
boring-cyborg[bot] commented on pull request #8677: URL: https://github.com/apache/airflow/pull/8677#issuecomment-623134852 Awesome work, congrats on your first merged pull request! This is an automated message from the

[GitHub] [airflow] pujaji commented on issue #8122: Guide about authorization and permission

2020-05-03 Thread GitBox
pujaji commented on issue #8122: URL: https://github.com/apache/airflow/issues/8122#issuecomment-623128120 Got it Totally! Thanks This is

  1   2   >