[GitHub] [airflow] sahil-b-shah opened a new issue #9732: DockerOperator does not support wait timeout for container

2020-07-08 Thread GitBox 


sahil-b-shah opened a new issue #9732:
URL: https://github.com/apache/airflow/issues/9732


   
   
   
   
   **Apache Airflow version**: 1.10.10
   
   
   **Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
   
   **Environment**: 
   
   - **Cloud provider or hardware configuration**: AWS
   - **OS** (e.g. from /etc/os-release): Debian GNU/Linux 10 (buster)
   - **Kernel** (e.g. `uname -a`): Linux Ubuntu SMP 2018 x86_64 GNU/Linux
   - **Install tools**: Python 3.6.11
   - **Others**:
   Docker version 18.03.0-ce
   docker==4.2.0
   
   **What happened**:
   Modified the `DockerOperator` to accept a `timeout` parameter that gets 
passed on to the `cli.wait` command (i.e. `result = 
self.cli.wait(self.container["Id"], timeout=self.timeout)` ). 
   
   The timeout does not work correctly. The container goes till full execution 
even if it's past the timeout window.
   
   
   
   **What you expected to happen**:
   
   The following commands work correctly when directly using the same docker 
python api on my local computer, which is Docker version 19.03.5. 
   
   ```
   from docker import APIClient
   cli = APIClient()
   container = cli.create_container("alpine", "sleep 45")
   cli.start(container["Id"])
   cli.wait(container["Id"], timeout=5)
   # urllib3.exceptions.ReadTimeoutError after 5 seconds
   ```
   
   
   
   **How to reproduce it**:
   
   Pass a `timeout` value in the call to `cli.wait` in L 260 of 
airflow/providers/docker/operators/docker.py
   
   
   
   
   **Anything else we need to know**:
   
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #9732: DockerOperator does not support wait timeout for container

2020-07-08 Thread GitBox 


boring-cyborg[bot] commented on issue #9732:
URL: https://github.com/apache/airflow/issues/9732#issuecomment-655913141


   Thanks for opening your first issue here! Be sure to follow the issue 
template!
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Fokko commented on pull request #9715: create CODE_OF_CONDUCT.md

2020-07-08 Thread GitBox 


Fokko commented on pull request #9715:
URL: https://github.com/apache/airflow/pull/9715#issuecomment-655912419


   I'm not against it, but why have another CoC next to the Apache one. I would 
suggest that we also refer to the Apache one.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated: Add read-only endpoints for DAG Model (#9045)

2020-07-08 Thread kamilbregula 
This is an automated email from the ASF dual-hosted git repository.

kamilbregula pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/master by this push:
 new 8b94ace  Add read-only endpoints for DAG Model (#9045)
8b94ace is described below

commit 8b94ace597f47e350161d799b6b45aad80f45ae4
Author: Kamil Breguła 
AuthorDate: Thu Jul 9 07:28:34 2020 +0200

Add read-only endpoints for DAG Model (#9045)

Co-authored-by: Tomek Urbaszek 
Co-authored-by: Tomek Urbaszek 
---
 airflow/api_connexion/endpoints/dag_endpoint.py|  35 --
 airflow/api_connexion/schemas/dag_schema.py|   1 +
 tests/api_connexion/endpoints/test_dag_endpoint.py | 130 -
 3 files changed, 151 insertions(+), 15 deletions(-)

diff --git a/airflow/api_connexion/endpoints/dag_endpoint.py 
b/airflow/api_connexion/endpoints/dag_endpoint.py
index 7cdeeb6..4f6aa2e 100644
--- a/airflow/api_connexion/endpoints/dag_endpoint.py
+++ b/airflow/api_connexion/endpoints/dag_endpoint.py
@@ -14,23 +14,30 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
-
 from flask import current_app
+from sqlalchemy import func
 
 from airflow import DAG
 from airflow.api_connexion.exceptions import NotFound
-# TODO(mik-laj): We have to implement it.
-# Do you want to help? Please look at:
-# * https://github.com/apache/airflow/issues/8128
-# * https://github.com/apache/airflow/issues/8138
-from airflow.api_connexion.schemas.dag_schema import dag_detail_schema
+from airflow.api_connexion.parameters import check_limit, format_parameters
+from airflow.api_connexion.schemas.dag_schema import (
+DAGCollection, dag_detail_schema, dag_schema, dags_collection_schema,
+)
+from airflow.models.dag import DagModel
+from airflow.utils.session import provide_session
 
 
-def get_dag():
+@provide_session
+def get_dag(dag_id, session):
 """
 Get basic information about a DAG.
 """
-raise NotImplementedError("Not implemented yet.")
+dag = session.query(DagModel).filter(DagModel.dag_id == 
dag_id).one_or_none()
+
+if dag is None:
+raise NotFound("DAG not found")
+
+return dag_schema.dump(dag)
 
 
 def get_dag_details(dag_id):
@@ -43,11 +50,19 @@ def get_dag_details(dag_id):
 return dag_detail_schema.dump(dag)
 
 
-def get_dags():
+@format_parameters({
+'limit': check_limit
+})
+@provide_session
+def get_dags(session, limit, offset=0):
 """
 Get all DAGs.
 """
-raise NotImplementedError("Not implemented yet.")
+dags = 
session.query(DagModel).order_by(DagModel.dag_id).offset(offset).limit(limit).all()
+
+total_entries = session.query(func.count(DagModel.dag_id)).scalar()
+
+return dags_collection_schema.dump(DAGCollection(dags=dags, 
total_entries=total_entries))
 
 
 def patch_dag():
diff --git a/airflow/api_connexion/schemas/dag_schema.py 
b/airflow/api_connexion/schemas/dag_schema.py
index aff859a..bae2228 100644
--- a/airflow/api_connexion/schemas/dag_schema.py
+++ b/airflow/api_connexion/schemas/dag_schema.py
@@ -89,4 +89,5 @@ class DAGCollectionSchema(Schema):
 
 dags_collection_schema = DAGCollectionSchema()
 dag_schema = DAGSchema()
+
 dag_detail_schema = DAGDetailSchema()
diff --git a/tests/api_connexion/endpoints/test_dag_endpoint.py 
b/tests/api_connexion/endpoints/test_dag_endpoint.py
index 6289b6f..1ba360f 100644
--- a/tests/api_connexion/endpoints/test_dag_endpoint.py
+++ b/tests/api_connexion/endpoints/test_dag_endpoint.py
@@ -19,11 +19,13 @@ import unittest
 from datetime import datetime
 
 import pytest
+from parameterized import parameterized
 
 from airflow import DAG
-from airflow.models import DagBag
+from airflow.models import DagBag, DagModel
 from airflow.models.serialized_dag import SerializedDagModel
 from airflow.operators.dummy_operator import DummyOperator
+from airflow.utils.session import provide_session
 from airflow.www import app
 from tests.test_utils.db import clear_db_dags, clear_db_runs, 
clear_db_serialized_dags
 
@@ -58,13 +60,41 @@ class TestDagEndpoint(unittest.TestCase):
 def tearDown(self) -> None:
 self.clean_db()
 
+@provide_session
+def _create_dag_models(self, count, session=None):
+for num in range(1, count + 1):
+dag_model = DagModel(
+dag_id=f"TEST_DAG_{num}",
+fileloc=f"/tmp/dag_{num}.py",
+schedule_interval="2 2 * * *"
+)
+session.add(dag_model)
+
 
 class TestGetDag(TestDagEndpoint):
-@pytest.mark.skip(reason="Not implemented yet")
 def test_should_response_200(self):
-response = self.client.get("/api/v1/dags/1/")
+self._create_dag_models(1)
+response = self.client.get("/api/v1/dags/TEST_DAG_1")
 assert response.status_code == 200
 
+current_response =

[GitHub] [airflow] mik-laj merged pull request #9045: Add read-only endpoints for DAG Model

2020-07-08 Thread GitBox 


mik-laj merged pull request #9045:
URL: https://github.com/apache/airflow/pull/9045


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj closed issue #8128: API Endpoints - Read-only - DAG

2020-07-08 Thread GitBox 


mik-laj closed issue #8128:
URL: https://github.com/apache/airflow/issues/8128


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #9731: Task instance null, execution date skipped

2020-07-08 Thread GitBox 


boring-cyborg[bot] commented on issue #9731:
URL: https://github.com/apache/airflow/issues/9731#issuecomment-655904657


   Thanks for opening your first issue here! Be sure to follow the issue 
template!
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] AyWa opened a new issue #9731: Task instance null, execution date skipped

2020-07-08 Thread GitBox 


AyWa opened a new issue #9731:
URL: https://github.com/apache/airflow/issues/9731


   **Apache Airflow version**: `1.10.9`
   **DB**:  `Postgre11.6` (aws rds)
   
   **Environment**:
   
   - **Cloud provider or hardware configuration**: `AWS BEANSTALK - DOCKER` in 
a `Local` executor
   
   **What happened**:
   
   In the UI, everything looks to be correctly run:
   
   https://user-images.githubusercontent.com/13608477/86998728-f3347c80-c1eb-11ea-962b-d4c09c9cacac.png;>
   
   However, we figure out that one execution date was missing (in our 
application). So I tried to find the reason why, and in the tree view, they was 
no circle for this date.
   So I ran `SELECT * FROM dag_run where 
execution_date='2020-05-25T00:00:00+00:00'` and there was no row too.
   
   After that I check the pool task, and by filter by date (filter by status 
not equal to success was showing nothing), I found:
   
   https://user-images.githubusercontent.com/13608477/86998972-7ce44a00-c1ec-11ea-9725-21433b181018.png;>
   
   So somehow, airflow UI is a bit misleading, because there was a task that 
failed to be run. But there was no easy way to figure out, without knowing the 
date or going to the database and running `SELECT * from task_instance where 
state is Null where  execution_date='2020-05-26T00:00:00+00:00';`
   
   ![Screen Shot 2020-07-09 at 1 24 37 
PM](https://user-images.githubusercontent.com/13608477/86999465-9e920100-c1ed-11ea-9dca-0f770acdb4cb.png)
   
   
   **What you expected to happen**:
   
   * I would expect to have an error or at least an indicator that a task 
instance are in null state or that a dag run was missing
   * I would expect to be able to filter `List Task Instance` by `NULL` or 
`None` status. However it was showing no result (if I was filtering by not 
equal to success it was empty too)
   * ideally I would expect to not have Null state, but If it is in the UI, I 
think it is acceptable
   
   **How to reproduce it**:
   
   Sadly I do not know how to reproduce, but I think by putting the data in the 
database we can at least "reproduce" the issue in the UI:
   
   - a dag run need to be empty `SELECT * FROM dag_run where 
execution_date='2020-05-25T00:00:00+00:00'`-> should give no row
   - task instance for this execution date should have an `Null` state like the 
screenshot shared. 
   
   Because we run in Elastic beanstalk, sometimes, container might be stop / 
recreated etc, so maybe it is related to that.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] zikun commented on pull request #9730: DbApiHook: Support kwargs in get_pandas_df

2020-07-08 Thread GitBox 


zikun commented on pull request #9730:
URL: https://github.com/apache/airflow/pull/9730#issuecomment-655886583


   Static check failed: 
   airflow/providers/google/cloud/hooks/bigquery.py:166: error: Signature of 
"get_pandas_df" incompatible with supertype "DbApiHook"
   
   Let me add kwargs in BigQueryHook as well



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] zikun opened a new pull request #9730: DbApiHook: Support kwargs in get_pandas_df

2020-07-08 Thread GitBox 


zikun opened a new pull request #9730:
URL: https://github.com/apache/airflow/pull/9730


   Support all parameters that are supported by pandas `read_sql` function: 
https://github.com/pandas-dev/pandas/blob/1.0.x/pandas/io/sql.py#L336-L345
   
   Closes #8468
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Target Github ISSUE in description if exists
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] tag nightly-master updated (6c15885 -> 0aea648)

2020-07-08 Thread github-bot 
This is an automated email from the ASF dual-hosted git repository.

github-bot pushed a change to tag nightly-master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


*** WARNING: tag nightly-master was modified! ***

from 6c15885  (commit)
  to 0aea648  (commit)
from 6c15885  generate go client from openapi spec (#9502)
 add ecce1ac  [AIRFLOW-] Remove unnecessary docstring in 
AWSAthenaOperator
 add c713d92  Add health API endpoint  (#8144) (#9277)
 add 564192c  Add AWS StepFunctions integrations to the aws provider (#8749)
 add 23f80f3  Move gcs & wasb task handlers to their respective provider 
packages (#9714)
 add 07b8102  Allow AWSAthenaHook to get more than 1000/first page of 
results (#6075)
 add 7a4988a  Add Dag Runs CRUD endpoints (#9473)
 add c353fed  Make airflow/migrations/env.py Pylint Compatible (#9670)
 add 2f31b30  Get Airflow configs with sensitive data from Secret Backends 
(#9645)
 add dfe8337  YAML file supports extra json parameters (#9549)
 add 38a0b1d  fix grammar in prereq tasks gcp operator docs (#9728)
 add b26017d  Add The Climate Corporation to user list (#9726)
 add 576100b  Add Qingping Hou to committers list (#9725)
 add 47c9f75  Add new fantastic team member of Polidea. (#9724)
 add 9db1fa3  Error in description after deployment (#9723)
 add 0aea648  Skip one version of Python for each test.

No new revisions were added by this update.

Summary of changes:
 .github/workflows/ci.yml   |   6 +-
 README.md  |   3 +-
 UPDATING.md|   8 +
 .../api_connexion/endpoints/dag_run_endpoint.py|  65 -
 airflow/api_connexion/endpoints/health_endpoint.py |  31 +-
 airflow/api_connexion/openapi/v1.yaml  | 104 ---
 airflow/api_connexion/schemas/dag_run_schema.py|  23 +-
 .../schemas/{log_schema.py => health_schema.py}|  27 +-
 airflow/configuration.py   |  65 -
 airflow/migrations/env.py  |   2 +-
 airflow/providers/amazon/aws/hooks/athena.py   |  64 +++-
 .../providers/amazon/aws/hooks/step_function.py|  79 +
 airflow/providers/amazon/aws/operators/athena.py   |   5 +-
 .../step_function_get_execution_output.py  |  58 
 .../aws/operators/step_function_start_execution.py |  72 +
 .../amazon/aws/secrets/secrets_manager.py  |  20 +-
 .../{athena.py => step_function_execution.py}  |  53 ++--
 .../google/cloud}/log/gcs_task_handler.py  |   0
 airflow/providers/hashicorp/secrets/vault.py   |  22 +-
 .../providers/microsoft/azure/log}/__init__.py |   0
 .../microsoft/azure}/log/wasb_task_handler.py  |   0
 airflow/secrets/__init__.py|  37 ++-
 airflow/secrets/base_secrets.py|  17 +-
 airflow/secrets/local_filesystem.py|  11 +-
 airflow/secrets/metastore.py   |   9 +-
 airflow/utils/log/gcs_task_handler.py  | 177 +--
 airflow/utils/log/wasb_task_handler.py | 179 +---
 chart/templates/NOTES.txt  |   4 +-
 docs/autoapi_templates/index.rst   |   1 +
 .../operator/gcp/_partials/prerequisite_tasks.rst  |  10 +-
 docs/howto/set-config.rst  |  29 +-
 docs/howto/use-alternative-secrets-backend.rst |  18 +-
 docs/operators-and-hooks-ref.rst   |   7 +
 docs/project.rst   |   4 +-
 pylintrc   |   2 +-
 scripts/ci/pylint_todo.txt |   1 -
 .../endpoints/test_dag_run_endpoint.py | 323 -
 .../endpoints/test_health_endpoint.py  |  68 -
 tests/api_connexion/schemas/test_dag_run_schema.py | 150 +-
 .../schemas/test_health_schema.py} |  29 +-
 tests/deprecated_classes.py|   8 +
 tests/providers/amazon/aws/hooks/test_athena.py| 172 +++
 .../amazon/aws/hooks/test_step_function.py |  63 
 ... => test_step_function_get_execution_output.py} |  55 ++--
 .../test_step_function_start_execution.py  |  82 ++
 .../aws/sensors/test_step_function_execution.py| 107 +++
 tests/providers/hashicorp/secrets/test_vault.py|  31 ++
 tests/secrets/test_local_filesystem.py |  79 -
 tests/test_configuration.py|  50 +++-
 tests/test_project_structure.py|   3 +-
 50 files changed, 1734 insertions(+), 699 deletions(-)
 copy airflow/api_connexion/schemas/{log_schema.py => health_schema.py} (58%)
 create mode 100644 airflow/providers/amazon/aws/hooks/step_function.py
 create mode 100644 
airflow/providers/amazon/aws/operators/step_function_get_execution_output.py
 create mode 100644

[GitHub] [airflow] houqp commented on pull request #9725: Add Qingping Hou to committers list

2020-07-08 Thread GitBox 


houqp commented on pull request #9725:
URL: https://github.com/apache/airflow/pull/9725#issuecomment-655827817


   Sorry, late to the party, thank you all for taking care of it :D



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] morrme commented on issue #9708: Increase typing coverage

2020-07-08 Thread GitBox 


morrme commented on issue #9708:
URL: https://github.com/apache/airflow/issues/9708#issuecomment-655826410


   @mik-laj Thank you! I will try celery!



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on issue #9708: Increase typing coverage

2020-07-08 Thread GitBox 


mik-laj commented on issue #9708:
URL: https://github.com/apache/airflow/issues/9708#issuecomment-655825714


   These are small providers and this can be a good start to the adventure.
   
   - discord
   - dingding 
   - datadog 
   - cloudant
   - celery
   
   If you need help, you can ask @ephraimbuddy or @OmairK, or on the 
#newbie-question channel on [our Slack 
channel](https://apache-airflow-slack.herokuapp.com/).



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj edited a comment on issue #9708: Increase typing coverage

2020-07-08 Thread GitBox 


mik-laj edited a comment on issue #9708:
URL: https://github.com/apache/airflow/issues/9708#issuecomment-655825714


   These are small providers and this can be a good start to the adventure.
   
   - discord
   - dingding 
   - datadog 
   - cloudant
   - celery
   
   If you need help, you can ask @ephraimbuddy or @OmairK, or on the 
#newbie-questions channel on [our Slack 
channel](https://apache-airflow-slack.herokuapp.com/).



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] morrme edited a comment on issue #9708: Increase typing coverage

2020-07-08 Thread GitBox 


morrme edited a comment on issue #9708:
URL: https://github.com/apache/airflow/issues/9708#issuecomment-655824265


   I would like to help with this. I am new to the project, so I may need some 
guidance, if available. 
   I have no provider preference. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] morrme commented on issue #9708: Increase typing coverage

2020-07-08 Thread GitBox 


morrme commented on issue #9708:
URL: https://github.com/apache/airflow/issues/9708#issuecomment-655824265


   I would like to help with this. I am new to the project, so I may need some 
guidance, if available. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on issue #9708: Increase typing coverage

2020-07-08 Thread GitBox 


mik-laj commented on issue #9708:
URL: https://github.com/apache/airflow/issues/9708#issuecomment-655821620


   @rafyzg I assigned you to the ticket. Which provider is interesting for you?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #9544: Add metric for scheduling delay between first run task & expected start time

2020-07-08 Thread GitBox 


mik-laj commented on pull request #9544:
URL: https://github.com/apache/airflow/pull/9544#issuecomment-655818454


   Can you add this test case to avoid regression in number of queries?
   ```python
   @provide_session
   def test_process_dags_queries_count_after_finish_dag_run(self, session):
   with mock.patch.dict("os.environ", {
   "PERF_DAGS_COUNT": "3",
   "PERF_TASKS_COUNT": "20",
   "PERF_START_AGO": "1d",
   "PERF_SCHEDULE_INTERVAL": "16h",
   "PERF_SHAPE": "grid",
   }), conf_vars({
   ('scheduler', 'use_job_schedule'): 'True',
   }):
   dagbag = DagBag(dag_folder=ELASTIC_DAG_FILE, 
include_examples=False)
   processor = DagFileProcessor([], mock.MagicMock())
   
   # Create new DAG Runs
   with assert_queries_count(28):
   processor._process_dags(dagbag.dags.values())
   
   self.assertEqual(session.query(DagRun).count(), 3)
   self.assertEqual(session.query(DagRun).filter(DagRun.state == 
State.RUNNING).count(), 3)
   
   # No new DAG Run
   with assert_queries_count(19):
   processor._process_dags(dagbag.dags.values())
   
   self.assertEqual(session.query(DagRun).count(), 3)
   self.assertEqual(session.query(DagRun).filter(DagRun.state == 
State.RUNNING).count(), 3)
   
   session.query(TaskInstance).update({
   "state": State.SUCCESS,
   "start_date": timezone.utcnow(),
   "end_date": timezone.utcnow(),
   "duration": 0,
   })
   
   # Finish Dag Runs
   with assert_queries_count(19):
   processor._process_dags(dagbag.dags.values())
   
   self.assertEqual(session.query(DagRun).count(), 3)
   self.assertEqual(session.query(DagRun).filter(DagRun.state == 
State.RUNNING).count(), 0)
   
   # No new DAG Runs
   with assert_queries_count(7):
   processor._process_dags(dagbag.dags.values())
   
   self.assertEqual(session.query(DagRun).count(), 3)
   self.assertEqual(session.query(DagRun).filter(DagRun.state == 
State.RUNNING).count(), 0)@provide_session
   def test_process_dags_queries_count_after_finish_dag_run(self, session):
   with mock.patch.dict("os.environ", {
   "PERF_DAGS_COUNT": "3",
   "PERF_TASKS_COUNT": "20",
   "PERF_START_AGO": "1d",
   "PERF_SCHEDULE_INTERVAL": "16h",
   "PERF_SHAPE": "grid",
   }), conf_vars({
   ('scheduler', 'use_job_schedule'): 'True',
   }):
   dagbag = DagBag(dag_folder=ELASTIC_DAG_FILE, 
include_examples=False)
   processor = DagFileProcessor([], mock.MagicMock())
   
   # Create new DAG Runs
   with assert_queries_count(28):
   processor._process_dags(dagbag.dags.values())
   
   self.assertEqual(session.query(DagRun).count(), 3)
   self.assertEqual(session.query(DagRun).filter(DagRun.state == 
State.RUNNING).count(), 3)
   
   # No new DAG Run
   with assert_queries_count(19):
   processor._process_dags(dagbag.dags.values())
   
   self.assertEqual(session.query(DagRun).count(), 3)
   self.assertEqual(session.query(DagRun).filter(DagRun.state == 
State.RUNNING).count(), 3)
   
   session.query(TaskInstance).update({
   "state": State.SUCCESS,
   "start_date": timezone.utcnow(),
   "end_date": timezone.utcnow(),
   "duration": 0,
   })
   
   # Finish Dag Runs
   with assert_queries_count(19):
   processor._process_dags(dagbag.dags.values())
   
   self.assertEqual(session.query(DagRun).count(), 3)
   self.assertEqual(session.query(DagRun).filter(DagRun.state == 
State.RUNNING).count(), 0)
   
   # No new DAG Runs
   with assert_queries_count(7):
   processor._process_dags(dagbag.dags.values())
   
   self.assertEqual(session.query(DagRun).count(), 3)
   self.assertEqual(session.query(DagRun).filter(DagRun.state == 
State.RUNNING).count(), 0)
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] rafyzg commented on issue #9708: Increase typing coverage

2020-07-08 Thread GitBox 


rafyzg commented on issue #9708:
URL: https://github.com/apache/airflow/issues/9708#issuecomment-655813328


   Hi @mik-laj ,  I would love to help with one of the providers. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #9544: Add metric for scheduling delay between first run task & expected start time

2020-07-08 Thread GitBox 


mik-laj commented on pull request #9544:
URL: https://github.com/apache/airflow/pull/9544#issuecomment-655812648


   I have the answer to the 3rd question. We only test cases when DAG Run are 
still running.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on a change in pull request #9544: Add metric for scheduling delay between first run task & expected start time

2020-07-08 Thread GitBox 


mik-laj commented on a change in pull request #9544:
URL: https://github.com/apache/airflow/pull/9544#discussion_r451883526



##
File path: airflow/models/dagrun.py
##
@@ -411,6 +412,44 @@ def _are_premature_tis(
 return True
 return False
 
+@provide_session
+def _emit_true_scheduling_delay_stats_for_finished_state(self, 
session=None):
+"""
+This is a helper method to emit the true scheduling delay stats, which 
is defined as
+the time when the first task in DAG starts minus the expected DAG run 
datetime.
+This method will be used in the update_state method when the state of 
the DagRun
+is updated to a completed status (either success or failure). The 
method will find the first
+started task within the DAG and calculate the expected DagRun start 
time (based on
+dag.execution_date & dag.schedule_interval), and minus these two to 
get the delay.
+
+The emitted data may contains outlier (e.g. when the first task was 
cleared, so
+the second task's start_date will be used), but we can get ride of the 
the outliers
+on the stats side through the dashboards.
+
+Note, the stat will only be emitted if the DagRun is a scheduler 
triggered one
+(i.e. external_trigger is False).
+"""
+if self.state == State.RUNNING:
+return
+
+try:
+if self.external_trigger:
+return
+# Get the task that has the earliest start_date
+qry = session.query(TI).filter(

Review comment:
   ```suggestion
   qry = session.query(TI.start_date).filter(
   ```
   Do you need all attributes?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #9544: Add metric for scheduling delay between first run task & expected start time

2020-07-08 Thread GitBox 


mik-laj commented on pull request #9544:
URL: https://github.com/apache/airflow/pull/9544#issuecomment-655809178


   I started looking at this change and I have three questions.
   1. What is your average difference in value between 
`dagrun.schedule_delay.` and 
`dagrun..first_task_scheduling_delay`?
   1. From what I see, you fetch one task with one query. Have you tried to 
avoid it? It seems to me that you can have this data in your memory. Please 
look at: airflow/models/dagrun.py:295 (update_state method)
   1. Do you know why this change does not affect 
jobs.test_scheduler_job.TestDagFileProcessorQueriesCount? It seems to me that 
there should be a visible problem with too many queries, but for some reason 
this is not visible. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Updated] (AIRFLOW-5071) Thousand os Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?

2020-07-08 Thread Kaxil Naik (Jira) 


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kaxil Naik updated AIRFLOW-5071:

Fix Version/s: 1.10.12

> Thousand os Executor reports task instance X finished (success) although the 
> task says its queued. Was the task killed externally?
> --
>
> Key: AIRFLOW-5071
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5071
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG, scheduler
>Affects Versions: 1.10.3
>Reporter: msempere
>Priority: Critical
> Fix For: 1.10.12
>
> Attachments: image-2020-01-27-18-10-29-124.png, 
> image-2020-07-08-07-58-42-972.png
>
>
> I'm opening this issue because since I update to 1.10.3 I'm seeing thousands 
> of daily messages like the following in the logs:
>  
> ```
>  {{__init__.py:1580}} ERROR - Executor reports task instance  2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says 
> its queued. Was the task killed externally?
> {{jobs.py:1484}} ERROR - Executor reports task instance  2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says 
> its queued. Was the task killed externally?
> ```
> -And looks like this is triggering also thousand of daily emails because the 
> flag to send email in case of failure is set to True.-
> I have Airflow setup to use Celery and Redis as a backend queue service.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5071) Thousand os Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?

2020-07-08 Thread Kaxil Naik (Jira) 


[ 
https://issues.apache.org/jira/browse/AIRFLOW-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154068#comment-17154068
 ] 

Kaxil Naik commented on AIRFLOW-5071:
-

Thanks for the info [~sgrzemski].

[~potiuk] Yes let's tackle this for 1.10.12. Seems like it has been occurring 
since sometime.

> Thousand os Executor reports task instance X finished (success) although the 
> task says its queued. Was the task killed externally?
> --
>
> Key: AIRFLOW-5071
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5071
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG, scheduler
>Affects Versions: 1.10.3
>Reporter: msempere
>Priority: Critical
> Fix For: 1.10.12
>
> Attachments: image-2020-01-27-18-10-29-124.png, 
> image-2020-07-08-07-58-42-972.png
>
>
> I'm opening this issue because since I update to 1.10.3 I'm seeing thousands 
> of daily messages like the following in the logs:
>  
> ```
>  {{__init__.py:1580}} ERROR - Executor reports task instance  2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says 
> its queued. Was the task killed externally?
> {{jobs.py:1484}} ERROR - Executor reports task instance  2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says 
> its queued. Was the task killed externally?
> ```
> -And looks like this is triggering also thousand of daily emails because the 
> flag to send email in case of failure is set to True.-
> I have Airflow setup to use Celery and Redis as a backend queue service.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9729: Increase typing coverage for Apache provider package

2020-07-08 Thread GitBox 


ephraimbuddy commented on a change in pull request #9729:
URL: https://github.com/apache/airflow/pull/9729#discussion_r451870163



##
File path: airflow/hooks/base_hook.py
##
@@ -21,7 +21,7 @@
 from typing import List

Review comment:
   ```suggestion
   from typing import Any, List
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil edited a comment on issue #9713: Looping issue using Hashicorp Vault

2020-07-08 Thread GitBox 


kaxil edited a comment on issue #9713:
URL: https://github.com/apache/airflow/issues/9713#issuecomment-655796847


   Currently there is no built-in retry strategy to prevent the issue you 
mentioned
   
   I am happy for you to take on this ticket and implement a strategy to 
prevent this kind of issue.
   
   Assigned the ticket to you, let me know how I can help.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on issue #9713: Looping issue using Hashicorp Vault

2020-07-08 Thread GitBox 


kaxil commented on issue #9713:
URL: https://github.com/apache/airflow/issues/9713#issuecomment-655796847


   I am happy for you to take on this ticket and implement a strategy to 
prevent this kind of issue



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on a change in pull request #9729: Increase typing coverage for Apache provider package

2020-07-08 Thread GitBox 


mik-laj commented on a change in pull request #9729:
URL: https://github.com/apache/airflow/pull/9729#discussion_r451865773



##
File path: airflow/utils/decorators.py
##
@@ -21,13 +21,14 @@
 import os
 from copy import copy
 from functools import wraps
+from typing import Any, Callable, Dict
 
 from airflow.exceptions import AirflowException
 
 signature = inspect.signature
 
 
-def apply_defaults(func):
+def apply_defaults(func: Callable[..., Any]) -> Any:

Review comment:
   This is a bigger challenge. mypy is not too smart to handle this 
decorator correctly. Please look at: 
   https://github.com/apache/airflow/pull/8145
   https://github.com/apache/airflow/tree/master/airflow/mypy
   https://github.com/python/mypy/issues/3157





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy commented on a change in pull request #9729: Increase typing coverage for Apache provider package

2020-07-08 Thread GitBox 


ephraimbuddy commented on a change in pull request #9729:
URL: https://github.com/apache/airflow/pull/9729#discussion_r451863059



##
File path: airflow/utils/decorators.py
##
@@ -21,13 +21,14 @@
 import os
 from copy import copy
 from functools import wraps
+from typing import Any, Callable, Dict
 
 from airflow.exceptions import AirflowException
 
 signature = inspect.signature
 
 
-def apply_defaults(func):
+def apply_defaults(func: Callable[..., Any]) -> Any:

Review comment:
   ```suggestion
   def apply_defaults(func: Callable[..., Any]) -> Callable[..., Any]:
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on pull request #9715: create CODE_OF_CONDUCT.md

2020-07-08 Thread GitBox 


kaxil commented on pull request #9715:
URL: https://github.com/apache/airflow/pull/9715#issuecomment-655791145


   The only thing I see missing in this PR's COC is section around Reporting 
Guidelines which ASF COC covers 
(https://www.apache.org/foundation/policies/conduct#reporting-guidelines)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vanka56 commented on a change in pull request #9472: Add drop_partition functionality for HiveMetastoreHook

2020-07-08 Thread GitBox 


vanka56 commented on a change in pull request #9472:
URL: https://github.com/apache/airflow/pull/9472#discussion_r451859081



##
File path: tests/providers/apache/hive/hooks/test_hive.py
##
@@ -383,6 +383,10 @@ def test_table_exists(self):
 self.hook.table_exists(str(random.randint(1, 1)))
 )
 
+def test_drop_partition(self):
+self.assertTrue(self.hook.drop_partitions(self.table, db=self.database,
+  part_vals=[DEFAULT_DATE_DS]))
+

Review comment:
   Sounds fair. Let me get this changed :)





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on a change in pull request #9729: Increase typing coverage for Apache provider package

2020-07-08 Thread GitBox 


mik-laj commented on a change in pull request #9729:
URL: https://github.com/apache/airflow/pull/9729#discussion_r451859421



##
File path: airflow/hooks/base_hook.py
##
@@ -82,6 +82,6 @@ def get_hook(cls, conn_id: str) -> "BaseHook":
 connection = cls.get_connection(conn_id)
 return connection.get_hook()
 
-def get_conn(self):
+def get_conn(self) -> None:

Review comment:
   ```suggestion
   def get_conn(self) -> Any:
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dossett commented on a change in pull request #9079: [8970] Improve KubernetesPodOperator guide

2020-07-08 Thread GitBox 


dossett commented on a change in pull request #9079:
URL: https://github.com/apache/airflow/pull/9079#discussion_r451814841



##
File path: docs/howto/operator/kubernetes.rst
##
@@ -22,150 +22,96 @@
 KubernetesPodOperator
 =
 
+The 
:class:`~airflow.providers.cncf.kubernetes.operators.kubernetes_pod.KubernetesPodOperator`
 allows
+you to create and run Pods on a Kubernetes cluster.
+
+.. contents::
+  :depth: 1
+  :local:
+
 .. note::
   If you use `Google Kubernetes Engine 
`__, consider
   using the
   :ref:`GKEStartPodOperator ` operator as 
it

Review comment:
   Some description that `GKEStartPodOperator` extends 
`KubernetesPodOperator` and that therefore the rest of this documentation is 
still applicable would be really helpful.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ephraimbuddy opened a new pull request #9729: Increase typing coverage for Apache provider package

2020-07-08 Thread GitBox 


ephraimbuddy opened a new pull request #9729:
URL: https://github.com/apache/airflow/pull/9729


   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [ ] Description above provides context of the change
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Target Github ISSUE in description if exists
   - [ ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated: Skip one version of Python for each test.

2020-07-08 Thread potiuk 
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/master by this push:
 new 0aea648  Skip one version of Python for each test.
0aea648 is described below

commit 0aea648b4bf956b07a55007ebd4af4025decd6cb
Author: James Timmins 
AuthorDate: Wed Jul 8 13:26:07 2020 -0700

Skip one version of Python for each test.

Skip one version of Python for each test.
---
 .github/workflows/ci.yml | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 8ea01a3..6818c81 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -243,7 +243,7 @@ ${{ 
hashFiles('requirements/requirements-python${{matrix.python-version}}.txt')
 needs: [static-checks-1, static-checks-2, trigger-tests]
 strategy:
   matrix:
-python-version: [3.6, 3.7, 3.8]
+python-version: [3.6, 3.7]
 postgres-version: [9.6, 10]
 test-type: [Core, Integration]
   fail-fast: false
@@ -275,7 +275,7 @@ ${{ 
hashFiles('requirements/requirements-python${{matrix.python-version}}.txt')
 needs: [static-checks-1, static-checks-2, trigger-tests]
 strategy:
   matrix:
-python-version: [3.6, 3.7, 3.8]
+python-version: [3.7, 3.8]
 mysql-version: [5.7]
 test-type: [Core, Integration]
   fail-fast: false
@@ -307,7 +307,7 @@ ${{ 
hashFiles('requirements/requirements-python${{matrix.python-version}}.txt')
 needs: [static-checks-1, static-checks-2, trigger-tests]
 strategy:
   matrix:
-python-version: [3.6, 3.7, 3.8]
+python-version: [3.6, 3.8]
 test-type: [Core, Integration]
   fail-fast: false
 env:

[GitHub] [airflow] potiuk merged pull request #9716: Remove 3.7 tests for MySQL, Postgres, and Sqlite.

2020-07-08 Thread GitBox 


potiuk merged pull request #9716:
URL: https://github.com/apache/airflow/pull/9716


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #9723: Error in description after deployment

2020-07-08 Thread GitBox 


boring-cyborg[bot] commented on pull request #9723:
URL: https://github.com/apache/airflow/pull/9723#issuecomment-655737425


   Awesome work, congrats on your first merged pull request!
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk merged pull request #9723: Error in description after deployment

2020-07-08 Thread GitBox 


potiuk merged pull request #9723:
URL: https://github.com/apache/airflow/pull/9723


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated (47c9f75 -> 9db1fa3)

2020-07-08 Thread potiuk 
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 47c9f75  Add new fantastic team member of Polidea. (#9724)
 add 9db1fa3  Error in description after deployment (#9723)

No new revisions were added by this update.

Summary of changes:
 chart/templates/NOTES.txt | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

[airflow] branch master updated (b26017d -> 576100b)

2020-07-08 Thread potiuk 
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from b26017d  Add The Climate Corporation to user list (#9726)
 add 576100b  Add Qingping Hou to committers list (#9725)

No new revisions were added by this update.

Summary of changes:
 docs/project.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

[airflow] branch master updated: Add new fantastic team member of Polidea. (#9724)

2020-07-08 Thread potiuk 
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/master by this push:
 new 47c9f75  Add new fantastic team member of Polidea. (#9724)
47c9f75 is described below

commit 47c9f75d55c0b9e7d31fd7cff797b3a987413a2b
Author: Kamil Breguła 
AuthorDate: Wed Jul 8 22:22:32 2020 +0200

Add new fantastic team member of Polidea. (#9724)
---
 README.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/README.md b/README.md
index 1729929..42a1972 100644
--- a/README.md
+++ b/README.md
@@ -503,7 +503,7 @@ Currently **officially** using Airflow:
 1. [Plaid](https://www.plaid.com/) [[@plaid](https://github.com/plaid), 
[@AustinBGibbons](https://github.com/AustinBGibbons) & 
[@jeeyoungk](https://github.com/jeeyoungk)]
 1. [Playbuzz](https://www.playbuzz.com/) 
[[@clintonboys](https://github.com/clintonboys) & 
[@dbn](https://github.com/dbn)]
 1. [PMC](https://pmc.com/) [[@andrewm4894](https://github.com/andrewm4894)]
-1. [Polidea](https://www.polidea.com/) [[@potiuk](https://github.com/potiuk), 
[@mschickensoup](https://github.com/mschickensoup), 
[@mik-laj](https://github.com/mik-laj), 
[@turbaszek](https://github.com/turbaszek), 
[@michalslowikowski00](https://github.com/michalslowikowski00), 
[@olchas](https://github.com/olchas)]
+1. [Polidea](https://www.polidea.com/) [[@potiuk](https://github.com/potiuk), 
[@mschickensoup](https://github.com/mschickensoup), 
[@mik-laj](https://github.com/mik-laj), 
[@turbaszek](https://github.com/turbaszek), 
[@michalslowikowski00](https://github.com/michalslowikowski00), 
[@olchas](https://github.com/olchas)], [@debek](https://github.com/debek)
 1. [Poshmark](https://www.poshmark.com)
 1. [Postmates](http://www.postmates.com) 
[[@syeoryn](https://github.com/syeoryn)]
 1. [Premise](http://www.premise.com) 
[[@jmccallum-premise](https://github.com/jmccallum-premise)]

[GitHub] [airflow] potiuk merged pull request #9725: Add Qingping Hou to committers list

2020-07-08 Thread GitBox 


potiuk merged pull request #9725:
URL: https://github.com/apache/airflow/pull/9725


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk merged pull request #9724: Add new fantastic team member of Polidea

2020-07-08 Thread GitBox 


potiuk merged pull request #9724:
URL: https://github.com/apache/airflow/pull/9724


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dossett commented on a change in pull request #9079: [8970] Improve KubernetesPodOperator guide

2020-07-08 Thread GitBox 


dossett commented on a change in pull request #9079:
URL: https://github.com/apache/airflow/pull/9079#discussion_r451802108



##
File path: docs/howto/operator/kubernetes.rst
##
@@ -22,150 +22,96 @@
 KubernetesPodOperator
 =
 
+The 
:class:`~airflow.providers.cncf.kubernetes.operators.kubernetes_pod.KubernetesPodOperator`
 allows
+you to create and run Pods on a Kubernetes cluster.
+
+.. contents::
+  :depth: 1
+  :local:
+
 .. note::
   If you use `Google Kubernetes Engine 
`__, consider
   using the
   :ref:`GKEStartPodOperator ` operator as 
it

Review comment:
   This link does currently appear to be working.  should it be updated?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated (38a0b1d -> b26017d)

2020-07-08 Thread kaxilnaik 
This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 38a0b1d  fix grammar in prereq tasks gcp operator docs (#9728)
 add b26017d  Add The Climate Corporation to user list (#9726)

No new revisions were added by this update.

Summary of changes:
 README.md | 1 +
 1 file changed, 1 insertion(+)

[GitHub] [airflow] kaxil commented on pull request #9726: add The Climate Corporation to user list

2020-07-08 Thread GitBox 


kaxil commented on pull request #9726:
URL: https://github.com/apache/airflow/pull/9726#issuecomment-655734870


   Thanks for adding :)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #9726: add The Climate Corporation to user list

2020-07-08 Thread GitBox 


boring-cyborg[bot] commented on pull request #9726:
URL: https://github.com/apache/airflow/pull/9726#issuecomment-655734754


   Awesome work, congrats on your first merged pull request!
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil merged pull request #9726: add The Climate Corporation to user list

2020-07-08 Thread GitBox 


kaxil merged pull request #9726:
URL: https://github.com/apache/airflow/pull/9726


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated (dfe8337 -> 38a0b1d)

2020-07-08 Thread potiuk 
This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from dfe8337  YAML file supports extra json parameters (#9549)
 add 38a0b1d  fix grammar in prereq tasks gcp operator docs (#9728)

No new revisions were added by this update.

Summary of changes:
 docs/howto/operator/gcp/_partials/prerequisite_tasks.rst | 10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)

[GitHub] [airflow] mik-laj commented on pull request #9079: [8970] Improve KubernetesPodOperator guide

2020-07-08 Thread GitBox 


mik-laj commented on pull request #9079:
URL: https://github.com/apache/airflow/pull/9079#issuecomment-655724857


   I would be happy to add information that Kubernetes Executor is not required 
for Kubernetes Pod Operator. Two sentences about the behavior of 
KubernetesPodOperator when installing into clusters would also be helpful, but 
we can add them in a separate change. 
   
   My opinion is based on the Slack discussion. This is an important feedback 
on what users are looking for in the documentation.
   https://apache-airflow.slack.com/archives/CCV3FV9KL/p1594235643204100



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #9728: Fix grammar in prerequisite tasks for GCP operator documentation

2020-07-08 Thread GitBox 


boring-cyborg[bot] commented on pull request #9728:
URL: https://github.com/apache/airflow/pull/9728#issuecomment-655725020


   Awesome work, congrats on your first merged pull request!
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk merged pull request #9728: Fix grammar in prerequisite tasks for GCP operator documentation

2020-07-08 Thread GitBox 


potiuk merged pull request #9728:
URL: https://github.com/apache/airflow/pull/9728


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vuppalli opened a new pull request #9728: Fix grammar in prerequisite tasks for GCP operator documentation

2020-07-08 Thread GitBox 


vuppalli opened a new pull request #9728:
URL: https://github.com/apache/airflow/pull/9728


   ---
   This PR is an attempt to fix some grammar issues in prerequisite_tasks.rst 
for GCP operator documentation.
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Target Github ISSUE in description if exists
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vuppalli edited a comment on issue #9418: Deprecated AI Platform Operators and Runtimes in Example DAG

2020-07-08 Thread GitBox 


vuppalli edited a comment on issue #9418:
URL: https://github.com/apache/airflow/issues/9418#issuecomment-652672779


   Thank you for the information! I created a PR for this issue here: 
https://github.com/apache/airflow/pull/9727. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vuppalli opened a new pull request #9727: fix depr operators, versions, and typos

2020-07-08 Thread GitBox 


vuppalli opened a new pull request #9727:
URL: https://github.com/apache/airflow/pull/9727


   ---
   The AI platform example DAG had typos, was using older versions, and had 
deprecated operators. This PR is an attempt to fix these problems and 
corresponds to [this Github 
issue](https://github.com/apache/airflow/issues/9418). The relevant 
documentation will be updated soon (most likely next week), which corresponds 
to [this Github issue](https://github.com/apache/airflow/issues/8207). I look 
forward to getting feedback!
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Target Github ISSUE in description if exists
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vuppalli closed pull request #9618: Fix typos, older versions, and deprecated operators with AI platform example DAG

2020-07-08 Thread GitBox 


vuppalli closed pull request #9618:
URL: https://github.com/apache/airflow/pull/9618


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] HenryLinTw commented on issue #9717: airflow initdb failed on ORA-22858 altering xcom.value to BLOB with Oracle (ATP) DB as backend DB

2020-07-08 Thread GitBox 


HenryLinTw commented on issue #9717:
URL: https://github.com/apache/airflow/issues/9717#issuecomment-655703660


   From documentation: 
https://airflow.apache.org/docs/stable/howto/initialize-database.html
   
   it mentions "As Airflow was built to interact with its metadata using the 
great SqlAlchemy library, you should be able to use any database backend 
supported as a SqlAlchemy backend We recommend using MySQL or Postgres."
   
   that seems to be misleading to me then.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #9726: add The Climate Corporation to user list

2020-07-08 Thread GitBox 


boring-cyborg[bot] commented on pull request #9726:
URL: https://github.com/apache/airflow/pull/9726#issuecomment-655695716


   Congratulations on your first Pull Request and welcome to the Apache Airflow 
community! If you have any issues or are unsure about any anything please check 
our Contribution Guide 
(https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
   Here are some useful points:
   - Pay attention to the quality of your code (flake8, pylint and type 
annotations). Our [pre-commits]( 
https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks)
 will help you with that.
   - In case of a new feature add useful documentation (in docstrings or in 
`docs/` directory). Adding a new operator? Check this short 
[guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst)
 Consider adding an example DAG that shows how users should use it.
   - Consider using [Breeze 
environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for 
testing locally, it’s a heavy docker but it ships with a working Airflow and a 
lot of integrations.
   - Be patient and persistent. It might take some time to get a review or get 
the final approval from Committers.
   - Please follow [ASF Code of 
Conduct](https://www.apache.org/foundation/policies/conduct) for all 
communication including (but not limited to) comments on Pull Requests, Mailing 
list and Slack.
   - Be sure to read the [Airflow Coding style]( 
https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
   Apache Airflow is a community-driven project and together we are making it 
better .
   In case of doubts contact the developers at:
   Mailing List: d...@airflow.apache.org
   Slack: https://apache-airflow-slack.herokuapp.com/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jmelching opened a new pull request #9726: add The Climate Corporation to user list

2020-07-08 Thread GitBox 


jmelching opened a new pull request #9726:
URL: https://github.com/apache/airflow/pull/9726


   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [ x] Description above provides context of the change
   - [ x] Unit tests coverage for changes (not needed for documentation changes)
   - [ x] Target Github ISSUE in description if exists
   - [ x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ x] Relevant documentation is updated including usage instructions.
   - [ x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] turbaszek commented on pull request #9725: Add Qingping Hou to committers list

2020-07-08 Thread GitBox 


turbaszek commented on pull request #9725:
URL: https://github.com/apache/airflow/pull/9725#issuecomment-655687358


   @houqp would you mind taking a look?  



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] turbaszek opened a new pull request #9725: Add Qingping Hou to committers list

2020-07-08 Thread GitBox 


turbaszek opened a new pull request #9725:
URL: https://github.com/apache/airflow/pull/9725


   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Target Github ISSUE in description if exists
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj opened a new pull request #9724: Add new fantastic team member of Polidea.

2020-07-08 Thread GitBox 


mik-laj opened a new pull request #9724:
URL: https://github.com/apache/airflow/pull/9724


   CC: @mschickensoup @debek 
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [X] Description above provides context of the change
   - [X] Unit tests coverage for changes (not needed for documentation changes)
   - [X] Target Github ISSUE in description if exists
   - [X] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [X] Relevant documentation is updated including usage instructions.
   - [X] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] debek edited a comment on pull request #9723: Error in description after deployment

2020-07-08 Thread GitBox 


debek edited a comment on pull request #9723:
URL: https://github.com/apache/airflow/pull/9723#issuecomment-655665824


   > template
   
   
   
   > @debek thanks for the description! What do you think about updating the 
title to "Add namespace flag in helm template notes"?
   
   I think that it is a good idea. I added a new commit.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] debek commented on pull request #9723: Error in description after deployment

2020-07-08 Thread GitBox 


debek commented on pull request #9723:
URL: https://github.com/apache/airflow/pull/9723#issuecomment-655665824


   > template
   
   
   
   > @debek thanks for the description! What do you think about updating the 
title to "Add namespace flag in helm template notes"?
   I think that it is a good idea. I added a new commit.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] turbaszek commented on pull request #9723: Error in description after deployment

2020-07-08 Thread GitBox 


turbaszek commented on pull request #9723:
URL: https://github.com/apache/airflow/pull/9723#issuecomment-655664004


   @debek thanks for the description! What do you think about updating the 
title to "Add namespace flag in helm template notes"? 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on a change in pull request #9723: Error in description after deployment

2020-07-08 Thread GitBox 


potiuk commented on a change in pull request #9723:
URL: https://github.com/apache/airflow/pull/9723#discussion_r451720394



##
File path: chart/templates/NOTES.txt
##
@@ -22,8 +22,8 @@ Your release is named {{ .Release.Name }}.
 
 You can now access your dashboard(s) by executing the following command(s) and 
visiting the corresponding port at localhost in your browser:
 
-Airflow dashboard:kubectl port-forward svc/{{ .Release.Name 
}}-webserver {{ .Values.ports.airflowUI }}:{{ .Values.ports.airflowUI }}
+Airflow dashboard:kubectl port-forward svc/{{ .Release.Name 
}}-webserver {{ .Values.ports.airflowUI }}:{{ .Values.ports.airflowUI }} -n 
airflow
 {{- if eq .Values.executor "CeleryExecutor"}}
-Flower dashboard: kubectl port-forward svc/{{ .Release.Name }}-flower 
{{ .Values.ports.flowerUI }}:{{ .Values.ports.flowerUI }}
+Flower dashboard: kubectl port-forward svc/{{ .Release.Name }}-flower 
{{ .Values.ports.flowerUI }}:{{ .Values.ports.flowerUI }} -n airflow

Review comment:
   ```suggestion
   Flower dashboard: kubectl port-forward svc/{{ .Release.Name 
}}-flower {{ .Values.ports.flowerUI }}:{{ .Values.ports.flowerUI }} --namespace 
airflow
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on a change in pull request #9723: Error in description after deployment

2020-07-08 Thread GitBox 


potiuk commented on a change in pull request #9723:
URL: https://github.com/apache/airflow/pull/9723#discussion_r451720254



##
File path: chart/templates/NOTES.txt
##
@@ -22,8 +22,8 @@ Your release is named {{ .Release.Name }}.
 
 You can now access your dashboard(s) by executing the following command(s) and 
visiting the corresponding port at localhost in your browser:
 
-Airflow dashboard:kubectl port-forward svc/{{ .Release.Name 
}}-webserver {{ .Values.ports.airflowUI }}:{{ .Values.ports.airflowUI }}
+Airflow dashboard:kubectl port-forward svc/{{ .Release.Name 
}}-webserver {{ .Values.ports.airflowUI }}:{{ .Values.ports.airflowUI }} -n 
airflow

Review comment:
   ```suggestion
   Airflow dashboard:kubectl port-forward svc/{{ .Release.Name 
}}-webserver {{ .Values.ports.airflowUI }}:{{ .Values.ports.airflowUI }} 
--namespace airflow
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] turbaszek commented on pull request #9723: Error in description after deployment

2020-07-08 Thread GitBox 


turbaszek commented on pull request #9723:
URL: https://github.com/apache/airflow/pull/9723#issuecomment-655659197


   @debek thanks for the PR! Could you please add some description to your PR? 
Meaningful commit messages and descriptions make code review easier  



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] tooptoop4 commented on pull request #3250: [WIP] Add dms raw

2020-07-08 Thread GitBox 


tooptoop4 commented on pull request #3250:
URL: https://github.com/apache/airflow/pull/3250#issuecomment-655654441


   @jzucker2 do u have any updates?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] debek opened a new pull request #9723: Error in description after deployment

2020-07-08 Thread GitBox 


debek opened a new pull request #9723:
URL: https://github.com/apache/airflow/pull/9723


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #9723: Error in description after deployment

2020-07-08 Thread GitBox 


boring-cyborg[bot] commented on pull request #9723:
URL: https://github.com/apache/airflow/pull/9723#issuecomment-655653141


   Congratulations on your first Pull Request and welcome to the Apache Airflow 
community! If you have any issues or are unsure about any anything please check 
our Contribution Guide 
(https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
   Here are some useful points:
   - Pay attention to the quality of your code (flake8, pylint and type 
annotations). Our [pre-commits]( 
https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks)
 will help you with that.
   - In case of a new feature add useful documentation (in docstrings or in 
`docs/` directory). Adding a new operator? Check this short 
[guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst)
 Consider adding an example DAG that shows how users should use it.
   - Consider using [Breeze 
environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for 
testing locally, it’s a heavy docker but it ships with a working Airflow and a 
lot of integrations.
   - Be patient and persistent. It might take some time to get a review or get 
the final approval from Committers.
   - Please follow [ASF Code of 
Conduct](https://www.apache.org/foundation/policies/conduct) for all 
communication including (but not limited to) comments on Pull Requests, Mailing 
list and Slack.
   - Be sure to read the [Airflow Coding style]( 
https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
   Apache Airflow is a community-driven project and together we are making it 
better .
   In case of doubts contact the developers at:
   Mailing List: d...@airflow.apache.org
   Slack: https://apache-airflow-slack.herokuapp.com/
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] zikun commented on issue #9610: Pod logs from KubernetesPodOperator occasionally get replaced with "Task is not able to run"

2020-07-08 Thread GitBox 


zikun commented on issue #9610:
URL: https://github.com/apache/airflow/issues/9610#issuecomment-655617057


   Similar issue in #9626



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #9722: Airflow can't import DAG in UI and logs, but manual DAG trigger works

2020-07-08 Thread GitBox 


boring-cyborg[bot] commented on issue #9722:
URL: https://github.com/apache/airflow/issues/9722#issuecomment-655607231


   Thanks for opening your first issue here! Be sure to follow the issue 
template!
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] BobasB opened a new issue #9722: Airflow can't import DAG in UI and logs, but manual DAG trigger works

2020-07-08 Thread GitBox 


BobasB opened a new issue #9722:
URL: https://github.com/apache/airflow/issues/9722


   Hi, I have a very strange and specific behaviour of Airflow on AWS EKS 
cluster after deploying Calico to enforce network policies.  I have also 
created AWS support case, but I also need support from Airflow team. I will be 
very appreciated for any help.
   **What happened**:
   I have Airflow set-up running as 2 k8s pods (Airflow webserver and 
scheduler). Both Airflow pods use git-sync sidecar container to get DAGs from 
git and store it at k8s `emptyDir` volume. All works well on fresh EKS cluster 
without errors. But at the moment of deploing Calico 
https://docs.aws.amazon.com/eks/latest/userguide/calico.html to EKS cluster all 
DAGs with local imports become broken. Airflow has default k8s Network policy 
which allow all ingress/egress traffic without restrictions, and Airflow UI is 
accessible. But in the Airflow there is a message `DAG "helloWorld" seems to be 
missing.` and Airflow webserver became to generate an error in the logs: 
   ```
   [2020-07-08 14:43:38,784] {__init__.py:51} INFO - Using executor 
SequentialExecutor  
│
   │ [2020-07-08 14:43:38,784] {dagbag.py:396} INFO - Filling up the DagBag 
from /usr/local/airflow/dags/repo   
  │
   │ [2020-07-08 14:43:38,785] {dagbag.py:225} DEBUG - Importing 
/usr/local/airflow/dags/repo/airflow_dags/dag_test.py   
 │
   │ [2020-07-08 14:43:39,016] {dagbag.py:239} ERROR - Failed to import: 
/usr/local/airflow/dags/repo/airflow_dags/dag_test.py   
 │
   │ Traceback (most recent call last): 

  │
   │   File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", 
line 236, in process_file   
   │
   │ m = imp.load_source(mod_name, filepath)

  │
   │   File "/usr/local/lib/python3.7/imp.py", line 171, in load_source 

  │
   │ module = _load(spec)   

  │
   │   File "", line 696, in _load 

  │
   │   File "", line 677, in _load_unlocked

  │
   │   File "", line 728, in exec_module  

  │
   │   File "", line 219, in 
_call_with_frames_removed   
│
   │   File "/usr/local/airflow/dags/repo/airflow_dags/dag_test.py", line 5, in 

  │
   │ from airflow_dags.common import DEFAULT_ARGS   

  │
   │ ModuleNotFoundError: No module named 'airflow_dags'
   ```
   
   The DAG itself consists of 2 files: `dag_test.py` and `common.py`. Content 
of the files are:
   `common.py`
   ```
   from datetime import datetime, timedelta
   
   DEFAULT_ARGS = {
   'owner': 'airflow',
   'depends_on_past': False,
   'start_date': datetime(2020, 3, 26),
   'retry_delay': timedelta(minutes=1),
   }
   ```
   
   `dag_test.py` 
   ```
   from airflow import DAG
   from airflow.operators.bash_operator import BashOperator
   
   from airflow_dags.common import DEFAULT_ARGS
   
   dag = DAG('helloWorld', schedule_interval='*/5 * * * *', 
default_args=DEFAULT_ARGS)
   
   t1 = BashOperator(
   task_id='task_1',
   bash_command='echo "Hello World from Task 1"; sleep 30',
   dag=dag
   )
   ```
   
   *What I have already tried at the webserver and scheduler pods*:
   - ssh to Airflow pod and enter Python shell. All imports work fine, for 
example:
   ```
   airflow@airflow-webserver-78bc695cc7-l7z9s:~$ pwd
   /usr/local/airflow
   airflow@airflow-webserver-78bc695cc7-l7z9s:~$ python
   Python 3.7.4 (default, Oct 17 2019, 06:10:02)
   [GCC 8.3.0] on linux
   Type "help", "copyright", "credits" or "license" for more information.
   >>> from airflow_dags.common import DEFAULT_ARGS
   >>> print(DEFAULT_ARGS)
   {'owner': 'airflow', 'depends_on_past': False, 'start_date': 
datetime.datetime(2020, 3, 26, 0, 0), 'retry_delay': 
datetime.timedelta(seconds=60)}
   >>>
   ```
   - from pod bash shell, I can execute airflow command and `list_tasks`, and 
DAG

[airflow] branch master updated: YAML file supports extra json parameters (#9549)

2020-07-08 Thread kamilbregula 
This is an automated email from the ASF dual-hosted git repository.

kamilbregula pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/master by this push:
 new dfe8337  YAML file supports extra json parameters (#9549)
dfe8337 is described below

commit dfe8337ca2d3ed173d9ecc112938271519792c40
Author: Vinay G B 
AuthorDate: Wed Jul 8 20:59:49 2020 +0530

YAML file supports extra json parameters (#9549)

Co-authored-by: Kamil Breguła 
Co-authored-by: Vinay 
Co-authored-by: Kamil Breguła 
---
 airflow/secrets/local_filesystem.py| 11 +++-
 docs/howto/use-alternative-secrets-backend.rst | 18 +++---
 tests/secrets/test_local_filesystem.py | 79 +-
 3 files changed, 97 insertions(+), 11 deletions(-)

diff --git a/airflow/secrets/local_filesystem.py 
b/airflow/secrets/local_filesystem.py
index 90f565c..0c246af 100644
--- a/airflow/secrets/local_filesystem.py
+++ b/airflow/secrets/local_filesystem.py
@@ -101,6 +101,7 @@ def _parse_yaml_file(file_path: str) -> Tuple[Dict[str, 
List[str]], List[FileSyn
 return {}, [FileSyntaxError(line_no=1, message="The file is empty.")]
 try:
 secrets = yaml.safe_load(content)
+
 except yaml.MarkedYAMLError as e:
 return {}, [FileSyntaxError(line_no=e.problem_mark.line, 
message=str(e))]
 if not isinstance(secrets, dict):
@@ -180,7 +181,7 @@ def _create_connection(conn_id: str, value: Any):
 if isinstance(value, str):
 return Connection(conn_id=conn_id, uri=value)
 if isinstance(value, dict):
-connection_parameter_names = get_connection_parameter_names()
+connection_parameter_names = get_connection_parameter_names() | 
{"extra_dejson"}
 current_keys = set(value.keys())
 if not current_keys.issubset(connection_parameter_names):
 illegal_keys = current_keys - connection_parameter_names
@@ -189,6 +190,14 @@ def _create_connection(conn_id: str, value: Any):
 f"The object have illegal keys: {illegal_keys_list}. "
 f"The dictionary can only contain the following keys: 
{connection_parameter_names}"
 )
+if "extra" in value and "extra_dejson" in value:
+raise AirflowException(
+"The extra and extra_dejson parameters are mutually exclusive. 
"
+"Please provide only one parameter."
+)
+if "extra_dejson" in value:
+value["extra"] = json.dumps(value["extra_dejson"])
+del value["extra_dejson"]
 
 if "conn_id" in current_keys and conn_id != value["conn_id"]:
 raise AirflowException(
diff --git a/docs/howto/use-alternative-secrets-backend.rst 
b/docs/howto/use-alternative-secrets-backend.rst
index 10ca35e..33e766e 100644
--- a/docs/howto/use-alternative-secrets-backend.rst
+++ b/docs/howto/use-alternative-secrets-backend.rst
@@ -90,12 +90,13 @@ Storing and Retrieving Connections
 If you have set ``connections_file_path`` as ``/files/my_conn.json``, then the 
backend will read the
 file ``/files/my_conn.json`` when it looks for connections.
 
-The file can be defined in ``JSON``, ``YAML`` or ``env`` format.
+The file can be defined in ``JSON``, ``YAML`` or ``env`` format. Depending on 
the format, the data should be saved as a URL or as a connection object.
+Any extra json parameters can be provided using keys like ``extra_dejson`` and 
``extra``.
+The key ``extra_dejson`` can be used to provide parameters as JSON object 
where as the key ``extra`` can be used in case of a JSON string.
+The keys ``extra`` and ``extra_dejson`` are mutually exclusive.
 
 The JSON file must contain an object where the key contains the connection ID 
and the value contains
-the definitions of one or more connections. The connection can be defined as a 
URI (string) or JSON object.
-For a guide about defining a connection as a URI, see:: 
:ref:`generating_connection_uri`.
-For a description of the connection object parameters see 
:class:`~airflow.models.connection.Connection`.
+the definitions of one or more connections. In this format, the connection can 
be defined as a URI (string) or JSON object.
 The following is a sample JSON file.
 
 .. code-block:: json
@@ -117,10 +118,7 @@ The following is a sample JSON file.
 }
 
 The YAML file structure is similar to that of a JSON. The key-value pair of 
connection ID and the definitions of one or more connections.
-The connection can be defined as a URI (string) or JSON object.
-For a guide about defining a connection as a URI, see:: 
:ref:`generating_connection_uri`.
-For a description of the connection object parameters see 
:class:`~airflow.models.connection.Connection`.
-The following is a sample YAML file.
+In this format, the connection can be defined as a URI (string) or JSON object.
 
 .. code-block:: yaml
 
@@ -137,6 +135,10 @@ The following is a

[GitHub] [airflow] mik-laj merged pull request #9549: YAML file supports extra json parameters

2020-07-08 Thread GitBox 


mik-laj merged pull request #9549:
URL: https://github.com/apache/airflow/pull/9549


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] klyusba opened a new issue #9721: InvalidToken error

2020-07-08 Thread GitBox 


klyusba opened a new issue #9721:
URL: https://github.com/apache/airflow/issues/9721


   **Apache Airflow version**: 1.10.10
   
   **Environment**: puckel/docker-airflow with celery executor
   
   **What happened**:
   
   PostgresHook inside PythonOperator causes cryptography.fernet.InvalidToken 
error.
   PostgresOperator causes no error in the same dag.
   
   **What you expected to happen**:
   
   PostgresHook causes no error within PythonOperator as it does within 
PostgresOperator 
   
   **How to reproduce it**:
   
   dag example
   from airflow import DAG
   from airflow.operators.python_operator import PythonOperator
   from airflow.hooks.postgres_hook import PostgresHook
   from airflow.operators.postgres_operator import PostgresOperator
   import datetime
   
   
   default_args = {
   "owner": "airflow",
   "start_date": datetime.datetime(2020, 3, 4),
   }
   
   dag = DAG(
   "log_test2", 
   default_args=default_args,
   schedule_interval=None
   )
   
   
   def test():
   h = PostgresHook(pg_conn_id="pg_dc_dwhmeta")
   res = h.get_records("select 1;")
   print(res)
   
   
   t1 = PostgresOperator(
   task_id="t1", 
   sql="select 1;", 
   postgres_conn_id="pg_dc_dwhmeta", 
   dag=dag
   )
   
   t2 = PythonOperator(
   task_id='t2',
   python_callable=test,
   dag=dag
   )
   
   t1>>t2
   
   
   
   
   **Anything else we need to know**:
   
   t1.log 
   [2020-07-08 13:49:05,134] {{taskinstance.py:669}} INFO - Dependencies all 
met for 
   [2020-07-08 13:49:05,333] {{taskinstance.py:669}} INFO - Dependencies all 
met for 
   [2020-07-08 13:49:05,333] {{taskinstance.py:879}} INFO - 
   

   [2020-07-08 13:49:05,333] {{taskinstance.py:880}} INFO - Starting attempt 1 
of 1
   [2020-07-08 13:49:05,333] {{taskinstance.py:881}} INFO - 
   

   [2020-07-08 13:49:05,471] {{taskinstance.py:900}} INFO - Executing 
 on 2020-07-08T13:48:49.898014+00:00
   [2020-07-08 13:49:05,477] {{standard_task_runner.py:53}} INFO - Started 
process 36152 to run task
   [2020-07-08 13:49:05,981] {{logging_mixin.py:112}} INFO - Running %s on host 
%s  
f7636f8a5dc5
   [2020-07-08 13:49:06,434] {{postgres_operator.py:62}} INFO - Executing: 
select 1;
   [2020-07-08 13:49:06,482] {{logging_mixin.py:112}} INFO - [2020-07-08 
13:49:06,482] {{base_hook.py:87}} INFO - Using connection to: id: 
pg_dc_dwhmeta. Host: 10.5.84.51, Port: 5432, Schema: dc_dwhmeta_dev, Login: 
tech_dwhetl, Password: , extra: None
   [2020-07-08 13:49:06,493] {{logging_mixin.py:112}} INFO - [2020-07-08 
13:49:06,493] {{dbapi_hook.py:174}} INFO - select 1;
   [2020-07-08 13:49:06,605] {{taskinstance.py:1065}} INFO - Marking task as 
SUCCESS.dag_id=log_test2, task_id=t1, execution_date=20200708T134849, 
start_date=20200708T134905, end_date=20200708T134906
   [2020-07-08 13:49:14,780] {{logging_mixin.py:112}} INFO - [2020-07-08 
13:49:14,780] {{local_task_job.py:103}} INFO - Task exited with return code 0
   
   
   t2.log 
   [2020-07-08 13:49:17,503] {{taskinstance.py:669}} INFO - Dependencies all 
met for 
   [2020-07-08 13:49:17,717] {{taskinstance.py:669}} INFO - Dependencies all 
met for 
   [2020-07-08 13:49:17,718] {{taskinstance.py:879}} INFO - 
   

   [2020-07-08 13:49:17,718] {{taskinstance.py:880}} INFO - Starting attempt 1 
of 1
   [2020-07-08 13:49:17,718] {{taskinstance.py:881}} INFO - 
   

   [2020-07-08 13:49:17,874] {{taskinstance.py:900}} INFO - Executing 
 on 2020-07-08T13:48:49.898014+00:00
   [2020-07-08 13:49:17,880] {{standard_task_runner.py:53}} INFO - Started 
process 36181 to run task
   [2020-07-08 13:49:18,385] {{logging_mixin.py:112}} INFO - Running %s on host 
%s  
f7636f8a5dc5
   [2020-07-08 13:49:18,803] {{taskinstance.py:1145}} ERROR - 
   Traceback (most recent call last):
 File 
"/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 
983, in _run_raw_task
   result = task_copy.execute(context=context)
 File 
"/usr/local/lib/python3.6/site-packages/airflow/operators/python_operator.py", 
line 113, in execute
   return_value = self.execute_callable()
 File 
"/usr/local/lib/python3.6/site-packages/airflow/operators/python_operator.py", 
line 118, in execute_callable
   return self.python_callable(*self.op_args, **self.op_kwargs)
 File "/usr/local/airflow/dags/log_test/log_test2.py", line 22, in test
   res = h.get_records("select 1;")
 File "/usr/local/lib/python3.6/site-packages/airflow/hooks/dbapi_hook.py", 
line 115, in get_records
   with closing(self.get_conn()) as conn:
 File 
"/usr/local/lib/python3.6/site-packages/airflow/hooks/postgres_hook.py", line

[GitHub] [airflow] boring-cyborg[bot] commented on issue #9721: InvalidToken error

2020-07-08 Thread GitBox 


boring-cyborg[bot] commented on issue #9721:
URL: https://github.com/apache/airflow/issues/9721#issuecomment-69904


   Thanks for opening your first issue here! Be sure to follow the issue 
template!
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (AIRFLOW-6786) Adding KafkaConsumerHook, KafkaProducerHook, and KafkaSensor

2020-07-08 Thread ASF GitHub Bot (Jira) 


[ 
https://issues.apache.org/jira/browse/AIRFLOW-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17153647#comment-17153647
 ] 

ASF GitHub Bot commented on AIRFLOW-6786:
-

dferguson992 commented on pull request #7407:
URL: https://github.com/apache/airflow/pull/7407#issuecomment-66027


   Working on it @haidaraM, the build process is taking longer than I expected



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Adding KafkaConsumerHook, KafkaProducerHook, and KafkaSensor
> 
>
> Key: AIRFLOW-6786
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6786
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: contrib, hooks
>Affects Versions: 1.10.9
>Reporter: Daniel Ferguson
>Assignee: Daniel Ferguson
>Priority: Minor
>
> Add the KafkaProducerHook.
>  Add the KafkaConsumerHook.
>  Add the KafkaSensor which listens to messages with a specific topic.
>  Related Issue:
>  #1311 (Pre-dates Jira Migration)
> Reminder to contributors:
> You must add an Apache License header to all new files
>  Please squash your commits when possible and follow the 7 rules of good Git 
> commits
>  I am new to the community, I am not sure the files are at the right place or 
> missing anything.
> The sensor could be used as the first node of a dag where the second node can 
> be a TriggerDagRunOperator. The messages are polled in a batch and the dag 
> runs are dynamically generated.
> Thanks!
> Note, as per denied PR [#1415|https://github.com/apache/airflow/pull/1415], 
> it is important to mention these integrations are not suitable for 
> low-latency/high-throughput/streaming. For reference, [#1415 
> (comment)|https://github.com/apache/airflow/pull/1415#issuecomment-484429806].
> Co-authored-by: Dan Ferguson 
> [dferguson...@gmail.com|mailto:dferguson...@gmail.com]
>  Co-authored-by: YuanfΞi Zhu



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] dferguson992 commented on pull request #7407: [AIRFLOW-6786] Add KafkaConsumerHook, KafkaProduerHook and KafkaSensor

2020-07-08 Thread GitBox 


dferguson992 commented on pull request #7407:
URL: https://github.com/apache/airflow/pull/7407#issuecomment-66027


   Working on it @haidaraM, the build process is taking longer than I expected



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #9720: how to use airflow with all extra by docker

2020-07-08 Thread GitBox 


potiuk commented on issue #9720:
URL: https://github.com/apache/airflow/issues/9720#issuecomment-655547447


   Pleas use slack on Github to ask questions like that - that's why your task 
was marked as invalid because this is not a place to ask questions. Generally - 
you have to rebuild the image with the right build args. See IMAGES.rst for 
details



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk closed issue #9720: how to use airflow with all extra by docker

2020-07-08 Thread GitBox 


potiuk closed issue #9720:
URL: https://github.com/apache/airflow/issues/9720


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] turbaszek commented on pull request #9712: Use namedtuple for TaskInstanceKeyType

2020-07-08 Thread GitBox 


turbaszek commented on pull request #9712:
URL: https://github.com/apache/airflow/pull/9712#issuecomment-655541034


   > LGTM overall, but I noticed that you usually refer to instances of 
`TaskInstanceKeyType` as `key`, but in airflow/jobs/backfill_job.py you use 
`ti_key` and `ti` in airflow/models/taskinstance.py. What would you say about 
introducing some convention for variable name referring to 
`TaskInstanceKeyType`, just like `TaskInstance` is usually referred to as `ti`? 
Personally, I like `ti_key`.
   
   @olchas I would propose to use `ti_key` and `tik` for short. WDYT?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Comment Edited] (AIRFLOW-5071) Thousand os Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?

2020-07-08 Thread Lindsay Portelli (Jira) 


[ 
https://issues.apache.org/jira/browse/AIRFLOW-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152897#comment-17152897
 ] 

Lindsay Portelli edited comment on AIRFLOW-5071 at 7/8/20, 1:53 PM:


Running 1.10.10 using kubernetes executor. Getting this error intermittently 
but is more frequent recently.


was (Author: lindsable):
Running 1.10.10 deployed on K8s using kubernetes executor. Getting this error 
intermittently but is more frequent recently. Looking at the logs it seems like 
the KubernetesJobWatcher is failing due to a Broken Pipe and isn't getting 
recreated.

> Thousand os Executor reports task instance X finished (success) although the 
> task says its queued. Was the task killed externally?
> --
>
> Key: AIRFLOW-5071
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5071
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: DAG, scheduler
>Affects Versions: 1.10.3
>Reporter: msempere
>Priority: Critical
> Attachments: image-2020-01-27-18-10-29-124.png, 
> image-2020-07-08-07-58-42-972.png
>
>
> I'm opening this issue because since I update to 1.10.3 I'm seeing thousands 
> of daily messages like the following in the logs:
>  
> ```
>  {{__init__.py:1580}} ERROR - Executor reports task instance  2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says 
> its queued. Was the task killed externally?
> {{jobs.py:1484}} ERROR - Executor reports task instance  2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says 
> its queued. Was the task killed externally?
> ```
> -And looks like this is triggering also thousand of daily emails because the 
> flag to send email in case of failure is set to True.-
> I have Airflow setup to use Celery and Redis as a backend queue service.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] geosmart opened a new issue #9720: how to use airflow with all extra by docker

2020-07-08 Thread GitBox 


geosmart opened a new issue #9720:
URL: https://github.com/apache/airflow/issues/9720


   my dockerfile like this
   ```dockerfile
   FROM apache/airflow:1.10.10-python3.6
   ```
   
   as airflow dockfile comment 
   ```md
   # This is a multi-segmented image. It actually contains two images:
   #
   # airflow-build-image  - there all airflow dependencies can be installed (and
   #built - for those dependencies that require
   #build essentials). Airflow is installed there with
   #--user switch so that all the dependencies are
   #installed to ${HOME}/.local
   #
   # main - this is the actual production image that is much
   #smaller because it does not contain all the build
   #essentials. Instead the ${HOME}/.local folder
   #is copied from the build-image - this way we have
   #only result of installation and we do not need
   #all the build essentials. This makes the image
   #much smaller.
   ```
   
   
   i want to use ldap,but the image doesnot install ldap3 module,
   
   so how can i use the apache/airflow:1.10.10-python3.6 with all extra ,
   just like `pip install airflow[all]`



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #9720: how to use airflow with all extra by docker

2020-07-08 Thread GitBox 


boring-cyborg[bot] commented on issue #9720:
URL: https://github.com/apache/airflow/issues/9720#issuecomment-655524333


   Thanks for opening your first issue here! Be sure to follow the issue 
template!
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on a change in pull request #9719: Pre-create Celery db result tables before running Celery worker

2020-07-08 Thread GitBox 


kaxil commented on a change in pull request #9719:
URL: https://github.com/apache/airflow/pull/9719#discussion_r451536099



##
File path: airflow/cli/commands/celery_command.py
##
@@ -112,6 +113,24 @@ def worker(args):
 log=args.log_file,
 )
 
+backend = celery_app.backend
+
+if hasattr(backend, 'ResultSession'):
+# Pre-create the database tables now, otherwise SQLA via Celery has a
+# race condition where it one of the subprocesses can die with "Table

Review comment:
   ```suggestion
   # race condition where one of the subprocesses can die with "Table
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb opened a new pull request #9719: Pre-create Celery db result tables before running Celery worker

2020-07-08 Thread GitBox 


ashb opened a new pull request #9719:
URL: https://github.com/apache/airflow/pull/9719


   Otherwise at large scale this can end up with some tasks failing as they
   try to create the result table at the same time.
   
   This was always possible before, just exceedingly rare, but in large
   scale performance testing where I create a lot of tasks quickly
   (especially in my HA testing) I hit this a few times.
   
   This is also only a problem for fresh installs/clean DBs, as once these
   tables exist the possible race goes away.
   
   This is the same fix from #8909, just for runtime, not test time.
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Target Github ISSUE in description if exists
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] subkanthi commented on issue #9134: Support Flink operator

2020-07-08 Thread GitBox 


subkanthi commented on issue #9134:
URL: https://github.com/apache/airflow/issues/9134#issuecomment-655501248


   I thought this was very similar to the SparkSubmit operator where it expects 
spark-submit to be in the path and takes the jar file as a parameter. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil merged pull request #9645: Get Airflow configs with sensitive data from Secret Backends

2020-07-08 Thread GitBox 


kaxil merged pull request #9645:
URL: https://github.com/apache/airflow/pull/9645


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated (c353fed -> 2f31b30)

2020-07-08 Thread kaxilnaik 
This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from c353fed  Make airflow/migrations/env.py Pylint Compatible (#9670)
 add 2f31b30  Get Airflow configs with sensitive data from Secret Backends 
(#9645)

No new revisions were added by this update.

Summary of changes:
 airflow/configuration.py   | 65 --
 .../amazon/aws/secrets/secrets_manager.py  | 20 ++-
 airflow/providers/hashicorp/secrets/vault.py   | 22 +++-
 airflow/secrets/__init__.py| 37 
 airflow/secrets/base_secrets.py| 17 +-
 airflow/secrets/metastore.py   |  9 ++-
 docs/howto/set-config.rst  | 29 +-
 tests/providers/hashicorp/secrets/test_vault.py| 31 +++
 tests/test_configuration.py| 50 +++--
 9 files changed, 245 insertions(+), 35 deletions(-)

[GitHub] [airflow] ashb commented on issue #9718: store_serialized_dags causes a pile up of open connections to the database backend with ~3000 DagRuns

2020-07-08 Thread GitBox 


ashb commented on issue #9718:
URL: https://github.com/apache/airflow/issues/9718#issuecomment-655472447


   Could you also please run `EXPLAIN ANALYZE` on that delete query in your DB?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on issue #9718: store_serialized_dags causes a pile up of open connections to the database backend with ~3000 DagRuns

2020-07-08 Thread GitBox 


ashb commented on issue #9718:
URL: https://github.com/apache/airflow/issues/9718#issuecomment-655471935


   Can you give us an example dag that shows this problem too please?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] amr-noureldin commented on issue #9718: store_serialized_dags causes a pile up of open connections to the database backend with ~3000 DagRuns

2020-07-08 Thread GitBox 


amr-noureldin commented on issue #9718:
URL: https://github.com/apache/airflow/issues/9718#issuecomment-655469482


   Sorry for the typo, this issue was encountered when running on Potgres 10



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on a change in pull request #9645: Get Airflow configs with sensitive data from Secret Backends

2020-07-08 Thread GitBox 


ashb commented on a change in pull request #9645:
URL: https://github.com/apache/airflow/pull/9645#discussion_r451481769



##
File path: airflow/configuration.py
##
@@ -495,6 +532,10 @@ def as_dict(
 set (True, default), or should the _cmd options be left as the
 command to run (False)
 :type include_cmds: bool
+:param include_secret: Should the result of calling any *_secret 
config be
+set (True, default), or should the _secret options be left as the

Review comment:
   Oh right, that's alright then.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] turbaszek commented on issue #8923: Google AdManager

2020-07-08 Thread GitBox 


turbaszek commented on issue #8923:
URL: https://github.com/apache/airflow/issues/8923#issuecomment-655469510


   @kweitzner @dephusluke @Handtaker23 do anyone of you would like work on this 
one? We are happy to help  



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on issue #9718: store_serialized_dags causes a pile up of open connections to the database backend with ~3000 DagRuns

2020-07-08 Thread GitBox 


kaxil commented on issue #9718:
URL: https://github.com/apache/airflow/issues/9718#issuecomment-655466707


   Can you also test it with Postgres >= 9.6 on a test instance and let us know 
if you still see the issue



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on issue #9718: store_serialized_dags causes a pile up of open connections to the database backend with ~3000 DagRuns

2020-07-08 Thread GitBox 


kaxil commented on issue #9718:
URL: https://github.com/apache/airflow/issues/9718#issuecomment-655465887


   A workaround would be to set `max_num_rendered_ti_fields_per_task` to 0 
(https://airflow.readthedocs.io/en/stable/configurations-ref.html#max-num-rendered-ti-fields-per-task)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil edited a comment on issue #9718: store_serialized_dags causes a pile up of open connections to the database backend with ~3000 DagRuns

2020-07-08 Thread GitBox 


kaxil edited a comment on issue #9718:
URL: https://github.com/apache/airflow/issues/9718#issuecomment-655465887


   A workaround if you want to use Dag Serialization would be to set 
`max_num_rendered_ti_fields_per_task` to 0 
(https://airflow.readthedocs.io/en/stable/configurations-ref.html#max-num-rendered-ti-fields-per-task)
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

  1   2   >