date:20200714



milton0825 commented on a change in pull request #9824:
URL: https://github.com/apache/airflow/pull/9824#discussion_r454785868



##
File path: airflow/models/dag.py
##
@@ -1134,16 +1134,18 @@ def clear(
 do_it = utils.helpers.ask_yesno(question)
 
 if do_it:
-clear_task_instances(tis,
- session,
- dag=self,
- )
-if reset_dag_runs:
-self.set_dag_runs_state(session=session,
-start_date=start_date,
-end_date=end_date,
-state=State.NONE,
-)
+clear_task_instances(
+tis,
+session,
+dag=self,
+activate_dag_runs=False,  # We will set DagRun state later.

Review comment:
   When `activate_dag_runs=True`, `DagRun` will be set to `RUNNING`





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] milton0825 commented on a change in pull request #9824: Change DAG.clear to take dag_run_state



milton0825 commented on a change in pull request #9824:
URL: https://github.com/apache/airflow/pull/9824#discussion_r454785655



##
File path: airflow/models/dag.py
##
@@ -1065,12 +1064,13 @@ def clear(
 instances = tis.all()
 for ti in instances:
 if ti.operator == ExternalTaskMarker.__name__:
+task: ExternalTaskMarker = cast(ExternalTaskMarker, 
self.get_task(ti.task_id))

Review comment:
   This is to fix mypy issue.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] milton0825 commented on a change in pull request #9824: Change DAG.clear to take dag_run_state



milton0825 commented on a change in pull request #9824:
URL: https://github.com/apache/airflow/pull/9824#discussion_r454785478



##
File path: airflow/models/dag.py
##
@@ -1026,26 +1025,26 @@ def clear(
 tis = tis.filter(TI.task_id.in_(self.task_ids))
 
 if include_parentdag and self.is_subdag:
-
-p_dag = self.parent_dag.sub_dag(
-task_regex=r"^{}$".format(self.dag_id.split('.')[1]),
-include_upstream=False,
-include_downstream=True)
-
-tis = tis.union(p_dag.clear(
-start_date=start_date, end_date=end_date,
-only_failed=only_failed,
-only_running=only_running,
-confirm_prompt=confirm_prompt,
-include_subdags=include_subdags,
-include_parentdag=False,
-reset_dag_runs=reset_dag_runs,
-get_tis=True,
-session=session,
-recursion_depth=recursion_depth,
-max_recursion_depth=max_recursion_depth,
-dag_bag=dag_bag
-))
+if self.parent_dag is not None:

Review comment:
   This is to fix mypy issue.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] zhbdesign opened a new issue #9826: Celery command failed



zhbdesign opened a new issue #9826:
URL: https://github.com/apache/airflow/issues/9826


   I use a cluster of four machine components. When I execute the task, the 
task has been distributed, but there will be errors for each machine. The log 
is as follows:
   
   [2020-07-15 11:25:46,471: ERROR/ForkPoolWorker-1] None
   [2020-07-15 11:25:46,582: ERROR/ForkPoolWorker-2] Task 
airflow.executors.celery_executor.execute_command[c29ab0dd-7049-4aeb-9023-cda45b9d3462]
 raised unexpected: AirflowException('Celery command failed',)
   Traceback (most recent call last):
 File 
"/usr/local/lib/python3.6/site-packages/airflow/executors/celery_executor.py", 
line 78, in execute_command
   close_fds=True, env=env)
 File "/usr/local/lib/python3.6/subprocess.py", line 291, in check_call
   raise CalledProcessError(retcode, cmd)
   subprocess.CalledProcessError: Command '['airflow', 'run', 
'user_MySql_2_ClickHouse_increment_srt_Activity', 
'm2ctask_Homework_SubmitActivity_Member_inc', 
'2020-07-15T11:25:35.177616+00:00', '--local', '--pool', 'default_pool', '-sd', 
'/opt/airflow/dags/MySql_2_ClickHouse_srt_Activity.py']' returned non-zero exit 
status 1.
   
   During handling of the above exception, another exception occurred:
   
   Traceback (most recent call last):
 File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 
412, in trace_task
   R = retval = fun(*args, **kwargs)
 File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 
704, in __protected_call__
   return self.run(*args, **kwargs)
 File 
"/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/celery.py", 
line 171, in _inner
   reraise(*exc_info)
 File "/usr/local/lib/python3.6/site-packages/sentry_sdk/_compat.py", line 
57, in reraise
   raise value
 File 
"/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/celery.py", 
line 166, in _inner
   return f(*args, **kwargs)
 File 
"/usr/local/lib/python3.6/site-packages/airflow/executors/celery_executor.py", 
line 83, in execute_command
   raise AirflowException('Celery command failed')
   airflow.exceptions.AirflowException: Celery command failed
   [2020-07-15 11:25:46,705: ERROR/ForkPoolWorker-1] Task 
airflow.executors.celery_executor.execute_command[efcd61c3-bae5-43a6-a2ba-ff584ee5a9e9]
 raised unexpected: AirflowException('Celery command failed',)
   Traceback (most recent call last):
 File 
"/usr/local/lib/python3.6/site-packages/airflow/executors/celery_executor.py", 
line 78, in execute_command
   close_fds=True, env=env)
 File "/usr/local/lib/python3.6/subprocess.py", line 291, in check_call
   raise CalledProcessError(retcode, cmd)
   subprocess.CalledProcessError: Command '['airflow', 'run', 
'user_MySql_2_ClickHouse_increment_srt_Activity', 
'MySql_2_ClickHouse_Activity_Category_inc', '2020-07-15T11:25:35.177616+00:00', 
'--local', '--pool', 'default_pool', '-sd', 
'/opt/airflow/dags/MySql_2_ClickHouse_srt_Activity.py']' returned non-zero exit 
status 1.
   
   
   The configuration file is as follows:
   
   
   [core]
   # The folder where your airflow pipelines live, most likely a
   # subfolder in a code repository. This path must be absolute.
   dags_folder = /opt/airflow/dags
   
   # The folder where airflow should store its log files
   # This path must be absolute
   base_log_folder = /opt/airflow/logs
   
   # Airflow can store logs remotely in AWS S3, Google Cloud Storage or Elastic 
Search.
   # Set this to True if you want to enable remote logging.
   remote_logging = False
   
   # Users must supply an Airflow connection id that provides access to the 
storage
   # location.
   remote_log_conn_id =
   remote_base_log_folder =
   encrypt_s3_logs = False
   
   # Logging level
   logging_level = INFO
   
   # Logging level for Flask-appbuilder UI
   fab_logging_level = INFO
   
   # Logging class
   # Specify the class that will specify the logging configuration
   # This class has to be on the python classpath
   # Example: logging_config_class = 
my.path.default_local_settings.LOGGING_CONFIG
   logging_config_class =
   
   # Flag to enable/disable Colored logs in Console
   # Colour the logs when the controlling terminal is a TTY.
   colored_console_log = True
   
   # Log format for when Colored logs is enabled
   colored_log_format = [%%(blue)s%%(asctime)s%%(reset)s] 
{%%(blue)s%%(filename)s:%%(reset)s%%(lineno)d} 
%%(log_color)s%%(levelname)s%%(reset)s - %%(log_color)s%%(message)s%%(reset)s
   colored_formatter_class = 
airflow.utils.log.colored_log.CustomTTYColoredFormatter
   
   # Format of Log line
   log_format = [%%(asctime)s] {%%(filename)s:%%(lineno)d} %%(levelname)s - 
%%(message)s
   simple_log_format = %%(asctime)s %%(levelname)s - %%(message)s
   
   # Log filename format
   log_filename_template = {{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ 
try_number }}.log
   log_processor_filename_template = {{ filename }}.log

[airflow] tag nightly-master updated (1a1a8db -> 5805a36)

2020-07-14 Thread github-bot

This is an automated email from the ASF dual-hosted git repository.

github-bot pushed a change to tag nightly-master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


*** WARNING: tag nightly-master was modified! ***

from 1a1a8db  (commit)
  to 5805a36  (commit)
from 1a1a8db  Update paths in .github/boring-cyborg.yml (#9799)
 add 3695624  Minor typo fix in OpenAPI specification (#9809)
 add c7858e8  Enable annotations to be added to the webserver service 
(#9776)
 add 98f06ea  Make airflow package type check compatible (#9791)
 add 468e950  Update README to add Py 3.8 in supported versions (#9804)
 add 0eb5020  Remove unnecessary comprehension (#9805)
 add 0a2acf0  Add type annotations for redis provider (#9815)
 add 593a0dd  Remove package.json and yarn.lock from the prod image (#9814)
 add 69f82e6  For now cloud tools are not needed in CI (#9818)
 add 4636fc6  Python 3.8.4 release breaks our builds (#9820)
 add ed5004c  Allow `replace` flag in gcs_to_gcs operator. (#9667)
 add a2c5389  Add kylin operator (#9149)
 add 5805a36  Fix SqlAlchemy-Flask failure with python 3.8.4 (#9821)

No new revisions were added by this update.

Summary of changes:
 CONTRIBUTING.rst   |  16 +-
 Dockerfile |   1 +
 Dockerfile.ci  |   2 +-
 INSTALL|  16 +-
 README.md  |  11 +-
 airflow/api_connexion/openapi/v1.yaml  |  18 +--
 airflow/executors/celery_executor.py   |   2 +-
 airflow/models/baseoperator.py |   4 +-
 airflow/operators/sql.py   |   2 +-
 .../providers/apache/kylin}/__init__.py|   0
 .../apache/kylin/example_dags}/__init__.py |   0
 .../apache/kylin/example_dags/example_kylin_dag.py | 136 +
 .../providers/apache/kylin}/hooks/__init__.py  |   0
 airflow/providers/apache/kylin/hooks/kylin.py  |  77 ++
 .../providers/apache/kylin/operators}/__init__.py  |   0
 .../providers/apache/kylin/operators/kylin_cube.py | 170 +
 .../providers/google/cloud/transfers/gcs_to_gcs.py |  19 +++
 airflow/providers/redis/hooks/redis.py |   2 +-
 airflow/providers/redis/operators/redis_publish.py |  12 +-
 airflow/providers/redis/sensors/redis_key.py   |   6 +-
 airflow/providers/redis/sensors/redis_pub_sub.py   |   6 +-
 airflow/{serialization/__init__.py => py.typed}|   3 +-
 airflow/utils/db.py|  11 ++
 airflow/utils/file.py  |   2 +-
 airflow/www/security.py|   2 +-
 backport_packages/import_all_provider_classes.py   |   4 +-
 breeze |   2 +-
 chart/README.md|   5 +-
 chart/templates/webserver/webserver-service.yaml   |   4 +
 chart/values.yaml  |   2 +
 docs/autoapi_templates/index.rst   |   4 +
 docs/operators-and-hooks-ref.rst   |   6 +
 requirements/requirements-python3.6.txt|  57 +++
 requirements/requirements-python3.7.txt|  55 +++
 requirements/requirements-python3.8.txt|  56 +++
 requirements/setup-3.6.md5 |   2 +-
 requirements/setup-3.7.md5 |   2 +-
 requirements/setup-3.8.md5 |   2 +-
 scripts/ci/in_container/entrypoint_ci.sh   |   8 +-
 .../ci/in_container/run_generate_requirements.sh   |   2 +-
 scripts/ci/libraries/_build_images.sh  |   8 +-
 scripts/ci/libraries/_initialization.sh|   4 +
 setup.py   |   8 +-
 tests/jobs/test_backfill_job.py|   4 +-
 tests/models/test_taskinstance.py  |   2 +-
 .../apache/kylin}/__init__.py  |   0
 .../apache/kylin/hooks}/__init__.py|   0
 tests/providers/apache/kylin/hooks/test_kylin.py   |  60 
 .../apache/kylin/operators}/__init__.py|   0
 .../apache/kylin/operators/test_kylin_cube.py  | 170 +
 .../google/cloud/transfers/test_gcs_to_gcs.py  |  15 ++
 51 files changed, 847 insertions(+), 153 deletions(-)
 copy {tests/providers/zendesk/hooks => 
airflow/providers/apache/kylin}/__init__.py (100%)
 copy {tests/providers/zendesk/hooks => 
airflow/providers/apache/kylin/example_dags}/__init__.py (100%)
 create mode 100644 
airflow/providers/apache/kylin/example_dags/example_kylin_dag.py
 copy {tests/providers/zendesk => 
airflow/providers/apache/kylin}/hooks/__init__.py (100%)
 create mode 100644 airflow/providers/apache/kylin/hooks/kylin.py
 copy {tests/providers/zendesk/hooks =>

[GitHub] [airflow] morrme commented on a change in pull request #9775: improve typing for datadog provider



morrme commented on a change in pull request #9775:
URL: https://github.com/apache/airflow/pull/9775#discussion_r454739504



##
File path: airflow/providers/datadog/sensors/datadog.py
##
@@ -39,15 +41,15 @@ class DatadogSensor(BaseSensorOperator):
 @apply_defaults
 def __init__(
 self,
-datadog_conn_id='datadog_default',
-from_seconds_ago=3600,
-up_to_seconds_from_now=0,
-priority=None,
-sources=None,
-tags=None,
+datadog_conn_id: str = 'datadog_default',
+from_seconds_ago: int = 3600,
+up_to_seconds_from_now: int = 0,
+priority: Optional[str] = None,
+sources: Optional[List[Any]] = None,
+tags: Optional[List[Any]] = None,
 response_check=None,

Review comment:
   i was thinking maybe `bool`





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (AIRFLOW-7014) Operator for Apache Kylin



[ 
https://issues.apache.org/jira/browse/AIRFLOW-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157798#comment-17157798
 ] 

ASF GitHub Bot commented on AIRFLOW-7014:
-

liuyonghengheng commented on pull request #9149:
URL: https://github.com/apache/airflow/pull/9149#issuecomment-658492312


   @potiuk Thank you very much！



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Operator for Apache Kylin 
> --
>
> Key: AIRFLOW-7014
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7014
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: executors, operators
>Affects Versions: 1.10.0
>Reporter: Shao Feng Shi
>Assignee: liuyongheng
>Priority: Major
>
> Apache Kylin is an analytial data warehouse for big data. Kylin provides a 
> set of RESTful API for user to trigger the data loading, and also run SQL 
> queries against the OLAP cubes in sub-seconds latency. We developed a 
> KylinOperator within Airflow, so that user can easily trigger Kylin with 
> other tasks (hive, spark, etc), and plan to contribute into Airflow.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] liuyonghengheng commented on pull request #9149: [AIRFLOW-7014] Add Apache Kylin operator



liuyonghengheng commented on pull request #9149:
URL: https://github.com/apache/airflow/pull/9149#issuecomment-658492312


   @potiuk Thank you very much！



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #9825: Improve type hinting to provider cloudant

boring-cyborg[bot] commented on pull request #9825:
URL: https://github.com/apache/airflow/pull/9825#issuecomment-658468606

Congratulations on your first Pull Request and welcome to the Apache Airflow
community! If you have any issues or are unsure about any anything please check
our Contribution Guide
(https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type
annotations). Our [pre-commits](
https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks)
will help you with that.
- In case of a new feature add useful documentation (in docstrings or in
`docs/` directory). Adding a new operator? Check this short
[guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst)
Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze
environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for
testing locally, it’s a heavy docker but it ships with a working Airflow and a
lot of integrations.
- Be patient and persistent. It might take some time to get a review or get
the final approval from Committers.
- Please follow [ASF Code of
Conduct](https://www.apache.org/foundation/policies/conduct) for all
communication including (but not limited to) comments on Pull Requests, Mailing
list and Slack.
- Be sure to read the [Airflow Coding style](
https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it
better .
In case of doubts contact the developers at:
Mailing List: d...@airflow.apache.org
Slack: https://apache-airflow-slack.herokuapp.com/

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] rafyzg opened a new pull request #9825: Improve type hinting to provider cloudant



rafyzg opened a new pull request #9825:
URL: https://github.com/apache/airflow/pull/9825


   Adding type annotations to cloudant provider as per #9708 
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Target Github ISSUE in description if exists
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] milton0825 opened a new pull request #9824: Change DAG.clear to take dag_run_state



milton0825 opened a new pull request #9824:
URL: https://github.com/apache/airflow/pull/9824


   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [ ] Description above provides context of the change
   - [ ] Unit tests coverage for changes (not needed for documentation changes)
   - [ ] Target Github ISSUE in description if exists
   - [ ] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [ ] Relevant documentation is updated including usage instructions.
   - [ ] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dimon222 edited a comment on issue #8212: Can't read S3 remote logs using Airflow 1.10.9



dimon222 edited a comment on issue #8212:
URL: https://github.com/apache/airflow/issues/8212#issuecomment-658444006


   @cmlad yes its the reason for sure. And HTTP vs HTTPS no difference.
   
   I randomly found potential fix in one of previously mentioned issues
   https://github.com/apache/airflow/issues/8164#issuecomment-629621571
   This works, prepending monkey patching in dagbag.py. What is invoking 
dagbag.py so early ? Not sure.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dimon222 commented on issue #8212: Can't read S3 remote logs using Airflow 1.10.9



dimon222 commented on issue #8212:
URL: https://github.com/apache/airflow/issues/8212#issuecomment-658444006


   @cmlad yes its the reason for sure. And HTTP vs HTTPS no difference.
   
   I randomly found potential fix in one of previously mentioned issues
   https://github.com/apache/airflow/issues/8164#issuecomment-629621571
   This works, adding monkey patching in dagbag.py. What is invoking dagbag.py 
so early ? Not sure.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on a change in pull request #9798: Guide for AI Platform (previously Machine Learning Engine) Operators



mik-laj commented on a change in pull request #9798:
URL: https://github.com/apache/airflow/pull/9798#discussion_r454671502



##
File path: docs/howto/operator/google/cloud/mlengine.rst
##
@@ -0,0 +1,217 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+
+
+
+Google Cloud AI Platform Operators
+==
+
+`Google Cloud AI Platform `__ (formerly 
known
+as ML Engine) can be used to train machine learning models at scale, host 
trained models
+in the cloud, and use models to make predictions for new data. AI Platform is 
a collection
+of tools for training, evaluating, and tuning machine learning models. AI 
Platform can also
+be used to deploy a trained model, make predictions, and manage various model 
versions.
+
+.. contents::
+  :depth: 1
+  :local:
+
+Prerequisite tasks
+^^
+
+.. include:: ../_partials/prerequisite_tasks.rst
+
+Launching a Job

Review comment:
   Yes. We should add a link to this guide in the class description





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vuppalli commented on a change in pull request #9798: Guide for AI Platform (previously Machine Learning Engine) Operators



vuppalli commented on a change in pull request #9798:
URL: https://github.com/apache/airflow/pull/9798#discussion_r454662915



##
File path: docs/howto/operator/google/cloud/mlengine.rst
##
@@ -0,0 +1,217 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+
+
+
+Google Cloud AI Platform Operators
+==
+
+`Google Cloud AI Platform `__ (formerly 
known
+as ML Engine) can be used to train machine learning models at scale, host 
trained models
+in the cloud, and use models to make predictions for new data. AI Platform is 
a collection
+of tools for training, evaluating, and tuning machine learning models. AI 
Platform can also
+be used to deploy a trained model, make predictions, and manage various model 
versions.
+
+.. contents::
+  :depth: 1
+  :local:
+
+Prerequisite tasks
+^^
+
+.. include:: ../_partials/prerequisite_tasks.rst
+
+Launching a Job

Review comment:
   Could you clarify what the reference should be to? Would you like me to 
add a reference of the guide in the class description?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on a change in pull request #9798: Guide for AI Platform (previously Machine Learning Engine) Operators



mik-laj commented on a change in pull request #9798:
URL: https://github.com/apache/airflow/pull/9798#discussion_r454659035



##
File path: docs/howto/operator/google/cloud/mlengine.rst
##
@@ -0,0 +1,217 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+
+
+
+Google Cloud AI Platform Operators
+==
+
+`Google Cloud AI Platform `__ (formerly 
known
+as ML Engine) can be used to train machine learning models at scale, host 
trained models
+in the cloud, and use models to make predictions for new data. AI Platform is 
a collection
+of tools for training, evaluating, and tuning machine learning models. AI 
Platform can also
+be used to deploy a trained model, make predictions, and manage various model 
versions.
+
+.. contents::
+  :depth: 1
+  :local:
+
+Prerequisite tasks
+^^
+
+.. include:: ../_partials/prerequisite_tasks.rst
+
+Launching a Job

Review comment:
   Oh. I forgot. Can you add references in the description of the guide to 
this section?
   See: 
   https://github.com/apache/airflow/issues/8894





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vuppalli commented on pull request #9798: Guide for AI Platform (previously Machine Learning Engine) Operators



vuppalli commented on pull request #9798:
URL: https://github.com/apache/airflow/pull/9798#issuecomment-658421169


   @mik-laj Thanks for the suggestion, I did the rebase. How does everything 
look now?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on a change in pull request #9822: Add note in Updating.md about the change in `run_as_user` default



kaxil commented on a change in pull request #9822:
URL: https://github.com/apache/airflow/pull/9822#discussion_r454635632



##
File path: UPDATING.md
##
@@ -1476,6 +1476,28 @@ auth_backend = airflow.api.auth.backend.default
 Since XCom values can contain pickled data, we would no longer allow adding or
 changing XCom values from the UI.
 
+### Default for `run_as_user` configured has been changed to 5 from 0

Review comment:
   FYI this is in Airflow 1.10.11 section





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on a change in pull request #9806: Add redoc docs to sphinx



kaxil commented on a change in pull request #9806:
URL: https://github.com/apache/airflow/pull/9806#discussion_r454629107



##
File path: docs/exts/sphinx_script_update.py
##
@@ -0,0 +1,113 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import hashlib
+import json
+import os
+import sys
+import tempfile
+from distutils.file_util import copy_file
+from typing import Dict
+
+import requests
+from sphinx.builders import html as builders
+from sphinx.util import logging
+
+log = logging.getLogger(__name__)
+
+
+def _gethash(string: str):
+hash_object = hashlib.sha256(string.encode())
+return hash_object.hexdigest()
+
+
+def _user_cache_dir(appname=None):
+"""
+Return full path to the user-specific cache dir for this application
+"""
+if sys.platform == "win32":
+# Windows has a complex procedure to download the App Dir directory 
because this directory can be
+# changed in window registry, so i use temporary directory for cache
+path = os.path.join(tempfile.gettempdir(), appname)
+elif sys.platform == 'darwin':
+path = os.path.expanduser('~/Library/Caches')
+else:
+path = os.getenv('XDG_CACHE_HOME', os.path.expanduser('~/.cache'))
+path = os.path.join(path, appname)
+return path
+
+
+def fetch_and_cache(script_url: str, output_filename: str):
+"""Fetch URL to local cache and returns path."""
+cache_key = _gethash(script_url)
+cache_dir = _user_cache_dir("redoc-doc")
+cache_metadata_filepath = os.path.join(cache_dir, "cache-metadata.json")
+cache_filepath = os.path.join(cache_dir, F"{cache_key}-{output_filename}")
+# Create cache directory
+os.makedirs(cache_dir, exist_ok=True)
+# Load cache metadata
+cache_metadata: Dict[str, str] = {}
+if os.path.exists(cache_metadata_filepath):
+try:
+with open(cache_metadata_filepath, "r") as cache_file:
+cache_metadata = json.load(cache_file)
+except json.JSONDecodeError:
+os.remove(cache_metadata_filepath)
+etag = cache_metadata.get(cache_key)
+
+# If we have a file and etag, check the fast path
+if os.path.exists(cache_filepath) and etag:
+res = requests.get(script_url, headers={"If-None-Match": etag})
+if res.status_code == 304:
+return cache_filepath
+
+# Slow patch
+res = requests.get(script_url)
+res.raise_for_status()
+
+with open(cache_filepath, "wb") as output_file:
+output_file.write(res.content)
+
+# Save cache metadata, if needed
+etag = res.headers.get('etag', None)
+if etag:
+cache_metadata[cache_key] = etag
+with open(cache_metadata_filepath, 'w') as cache_file:
+json.dump(cache_metadata, cache_file)
+
+return cache_filepath
+
+
+def build_finished(app, exception):
+"""Sphpix "build_finished" event handler."""

Review comment:
   ```suggestion
   """Sphinx "build_finished" event handler."""
   ```

##
File path: docs/exts/sphinx_script_update.py
##
@@ -0,0 +1,113 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import hashlib
+import json
+import os
+import sys
+import tempfile
+from distutils.file_util import copy_file
+from typing import Dict
+
+import requests
+from sphinx.builders import html as builders
+from sphinx.util import logging
+
+log = logging.getLogger(__name__)
+
+
+def _gethash(string: str):
+hash_object =

[airflow] branch master updated: Fix SqlAlchemy-Flask failure with python 3.8.4 (#9821)

2020-07-14 Thread kaxilnaik

This is an automated email from the ASF dual-hosted git repository.

kaxilnaik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/master by this push:
 new 5805a36  Fix SqlAlchemy-Flask failure with python 3.8.4 (#9821)
5805a36 is described below

commit 5805a363689d71dcdeb8eb5f82fd81d075b8847a
Author: Jarek Potiuk 
AuthorDate: Tue Jul 14 22:28:42 2020 +0200

Fix SqlAlchemy-Flask failure with python 3.8.4 (#9821)
---
 Dockerfile.ci   |  2 +-
 requirements/requirements-python3.6.txt |  4 ++--
 requirements/requirements-python3.7.txt |  4 ++--
 requirements/requirements-python3.8.txt |  4 ++--
 scripts/ci/libraries/_initialization.sh | 12 ++--
 5 files changed, 9 insertions(+), 17 deletions(-)

diff --git a/Dockerfile.ci b/Dockerfile.ci
index bff6bbf..ade6a70 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -218,7 +218,7 @@ RUN echo "Installing with extras: ${AIRFLOW_EXTRAS}."
 
 # By changing the CI build epoch we can force reinstalling Arflow from the 
current master
 # It can also be overwritten manually by setting the AIRFLOW_CI_BUILD_EPOCH 
environment variable.
-ARG AIRFLOW_CI_BUILD_EPOCH="1"
+ARG AIRFLOW_CI_BUILD_EPOCH="2"
 ENV AIRFLOW_CI_BUILD_EPOCH=${AIRFLOW_CI_BUILD_EPOCH}
 
 # In case of CI builds we want to pre-install master version of airflow 
dependencies so that
diff --git a/requirements/requirements-python3.6.txt 
b/requirements/requirements-python3.6.txt
index 792e908..3ecde7f 100644
--- a/requirements/requirements-python3.6.txt
+++ b/requirements/requirements-python3.6.txt
@@ -9,7 +9,7 @@ Flask-JWT-Extended==3.24.1
 Flask-Login==0.4.1
 Flask-OAuthlib==0.9.5
 Flask-OpenID==1.2.5
-Flask-SQLAlchemy==2.4.3
+Flask-SQLAlchemy==2.4.4
 Flask-WTF==0.14.3
 Flask==1.1.2
 GitPython==3.1.7
@@ -332,7 +332,7 @@ sentinels==1.0.0
 sentry-sdk==0.16.1
 setproctitle==1.1.10
 sh==1.13.1
-simple-salesforce==1.1.0
+simple-salesforce==1.10.1
 six==1.15.0
 slackclient==2.7.2
 smmap==3.0.4
diff --git a/requirements/requirements-python3.7.txt 
b/requirements/requirements-python3.7.txt
index a055ace..b7a0145 100644
--- a/requirements/requirements-python3.7.txt
+++ b/requirements/requirements-python3.7.txt
@@ -9,7 +9,7 @@ Flask-JWT-Extended==3.24.1
 Flask-Login==0.4.1
 Flask-OAuthlib==0.9.5
 Flask-OpenID==1.2.5
-Flask-SQLAlchemy==2.4.3
+Flask-SQLAlchemy==2.4.4
 Flask-WTF==0.14.3
 Flask==1.1.2
 GitPython==3.1.7
@@ -327,7 +327,7 @@ sentinels==1.0.0
 sentry-sdk==0.16.1
 setproctitle==1.1.10
 sh==1.13.1
-simple-salesforce==1.1.0
+simple-salesforce==1.10.1
 six==1.15.0
 slackclient==2.7.2
 smmap==3.0.4
diff --git a/requirements/requirements-python3.8.txt 
b/requirements/requirements-python3.8.txt
index d199958..b0d2773 100644
--- a/requirements/requirements-python3.8.txt
+++ b/requirements/requirements-python3.8.txt
@@ -9,7 +9,7 @@ Flask-JWT-Extended==3.24.1
 Flask-Login==0.4.1
 Flask-OAuthlib==0.9.5
 Flask-OpenID==1.2.5
-Flask-SQLAlchemy==2.4.3
+Flask-SQLAlchemy==2.4.4
 Flask-WTF==0.14.3
 Flask==1.1.2
 GitPython==3.1.7
@@ -326,7 +326,7 @@ sentinels==1.0.0
 sentry-sdk==0.16.1
 setproctitle==1.1.10
 sh==1.13.1
-simple-salesforce==1.1.0
+simple-salesforce==1.10.1
 six==1.15.0
 slackclient==2.7.2
 smmap==3.0.4
diff --git a/scripts/ci/libraries/_initialization.sh 
b/scripts/ci/libraries/_initialization.sh
index 6e9c5c3..d159792 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -21,17 +21,9 @@ function initialize_common_environment {
 # default python Major/Minor version
 PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:="3.6"}
 
+# python image version to use
+# shellcheck disable=SC2034
 PYTHON_BASE_IMAGE_VERSION=${PYTHON_MAJOR_MINOR_VERSION}
-if [[ ${PYTHON_BASE_IMAGE_VERSION} == "3.8" ]]; then
-# Temporary fix for Python 3.8.4 failing our tests
-# TODO: Remove after it gets fixed
-PYTHON_BASE_IMAGE_VERSION="3.8.3"
-echo ""
-echo " WARNING! Temporary switching to ${PYTHON_BASE_IMAGE_VERSION}"
-echo "  Until we find root cause of problem introduced by 
3.8.4"
-echo
-echo ""
-fi
 
 # extra flags passed to docker run for CI image
 # shellcheck disable=SC2034

[GitHub] [airflow] kaxil merged pull request #9821: Fixes sqlalchemy flask failure with python 3.8.4



kaxil merged pull request #9821:
URL: https://github.com/apache/airflow/pull/9821


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] sjmiller609 opened a new issue #9823: Optional allow-list for operator name in metrics



sjmiller609 opened a new issue #9823:
URL: https://github.com/apache/airflow/issues/9823


   
   
   **Description**
   
   
   
   I am seeking approval for a minor feature. We are using Airflow metrics, two 
of these metrics are airflow.operator_failures_.* and 
airflow.operator_successes_.*. The name of these metrics will include the 
operator name, for example airflow.operator_successes_PythonOperator.
   
   With regard to time series data, it is best practice to reduce the possible 
values for any given attribute in a metric. For example, we are recording a 
metric in our TSDB "airflow_operator_success" with one of the attributes as 
"operator". For example, here are some sample entries:
   
   ```
   airflow_operator_failures{operator="ExternalTaskSensor"} 5
   airflow_operator_failures{operator="GCSToPostgresOperator"} 10
   airflow_operator_failures{operator="PipedriveToCloudStorageOperator"} 2
   airflow_operator_failures{operator="PostgresOperator"} 24
   airflow_operator_failures{operator="PrometheusToGCSOperator"} 128
   airflow_operator_failures{operator="MyTopSecretOperator"} 676
   ```
   
   We have a concern that since Operators may be named by the airflow user, 
that there are 1) too many (infinite) options for the "operator" attribute, 
thereby causing performance issues in the TSDB and 2) the metrics system is 
exposing potentially sensitive information, for example "MyTopSecretOperator", 
which our operations team prefers to omit from our metrics system.
   
   With approval, I may contribute a change that will retain the same behavior 
by default, but optionally accept an allow-list in the airflow configuration in 
the form of a list of operator names. If provided, this list will be the only 
allowed values for operator name in the metrics airflow.operator_successes_.* 
and airflow.operator_failures_.*, with operators named in any other way falling 
into an "other" category. For example:
   ```
   airflow_operator_failures{operator="BashOperator"} 5
   airflow_operator_failures{operator="PythonOperator"} 10
   airflow_operator_failures{operator="Other"} 100
   ```
   
   **Use case / motivation**
   
   
   
   In our use case, we are persisting metrics data using Prometheus and the 
statsd exporter (translates statsd into prometheus metrics language). It is 
best practice for prometheus (and any TSDB) to have minimized 
[cardinality](https://www.robustperception.io/cardinality-is-key) on all 
metrics. We have found performance issues with regard to 
airflow.operator_failures_.*. One option is for us to drop all information 
about operator name from this metric, but we believe that such a feature in 
airflow as proposed here would allow more informative metrics while retaining 
reasonable cardinality (for example, include all operators included in Airflow 
by default).
   
   In general, this story is one step to making Airflow metrics be more 
prometheus-friendly. There are other metrics that could be considered to be 
made more prometheus-friendly, but I hope to constrain this issue to only the 
two mentioned.
   
   **Related Issues**
   
   
   
   none
   
   **Alternatives**
   
   It is also possible for this to be accomplished in the statsd-exporter 
project.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #9823: Optional allow-list for operator name in metrics



boring-cyborg[bot] commented on issue #9823:
URL: https://github.com/apache/airflow/issues/9823#issuecomment-658390329


   Thanks for opening your first issue here! Be sure to follow the issue 
template!
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #9806: Add redoc docs to sphinx



mik-laj commented on pull request #9806:
URL: https://github.com/apache/airflow/pull/9806#issuecomment-658387971


   I updated the PR:  He used this plugin and added the JS library download on 
demand  (with cache).
   Demo: https://valuable-carpenter.surge.sh/



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #9778: Enable setting the base url of the website



mik-laj commented on pull request #9778:
URL: https://github.com/apache/airflow/pull/9778#issuecomment-658386920


   @dimberman The base_url option adds a prefix to each address (including 
health check), so this change looks good. This option is implemented as WSGI 
middleware, so it affects all addresses. This option is not related to reverse 
proxy, but is often used together. We have a similar option - proxy_fix and it 
is closely related to the option and has a similar purpose, but a different way 
of working.
   Please look here: 
   https://github.com/apache/airflow/blob/master/tests/www/test_app.py
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] dimberman commented on pull request #9778: Enable setting the base url of the website



dimberman commented on pull request #9778:
URL: https://github.com/apache/airflow/pull/9778#issuecomment-658383793


   @vsimon I guess I'm a bit confused here, wouldn't that be testing the 
reverse proxy instead of airflow? Why would you need to access airflow via the 
reverse proxy for a health check?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on a change in pull request #9798: Guide for AI Platform (previously Machine Learning Engine) Operators



mik-laj commented on a change in pull request #9798:
URL: https://github.com/apache/airflow/pull/9798#discussion_r454601015



##
File path: CONTRIBUTING.rst
##
@@ -314,14 +314,14 @@ This is the full list of those extras:
   .. START EXTRAS HERE
 
 all_dbs, amazon, apache.atlas, apache_beam, apache.cassandra, apache.druid, 
apache.hdfs,
-apache.hive, apache.pinot, apache.webhdfs, async, atlas, aws, azure, 
cassandra, celery, cgroups,
-cloudant, cncf.kubernetes, dask, databricks, datadog, devel, devel_hadoop, 
doc, docker, druid,
-elasticsearch, exasol, facebook, gcp, gcp_api, github_enterprise, google, 
google_auth, grpc,
-hashicorp, hdfs, hive, jdbc, jira, kerberos, kubernetes, ldap, 
microsoft.azure, microsoft.mssql,
-microsoft.winrm, mongo, mssql, mysql, odbc, oracle, pagerduty, papermill, 
password, pinot, postgres,
-presto, qds, rabbitmq, redis, salesforce, samba, segment, sendgrid, sentry, 
singularity, slack,
-snowflake, spark, ssh, statsd, tableau, vertica, virtualenv, webhdfs, winrm, 
yandexcloud, all,
-devel_ci
+apache.hive, apache.kylin, apache.pinot, apache.webhdfs, async, atlas, aws, 
azure, cassandra,

Review comment:
   It looks like an unrelated change. Can you do a rebase?  
   https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#id8





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #9798: Guide for AI Platform (previously Machine Learning Engine) Operators



mik-laj commented on pull request #9798:
URL: https://github.com/apache/airflow/pull/9798#issuecomment-658375127


   Here is more info about quarantined tests: 
https://github.com/apache/airflow/blob/master/TESTING.rst#quarantined-tests
   CI Build/ Push Prod images, CI Build/ Push CI images,  CI Build/ Push CI 
images are always skipped on pull requests. We run it on master branch only.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vuppalli commented on pull request #9798: Guide for AI Platform (previously Machine Learning Engine) Operators



vuppalli commented on pull request #9798:
URL: https://github.com/apache/airflow/pull/9798#issuecomment-658373375


   @mik-laj Thanks so much for providing me slack link! The branch should be in 
sync with the Airflow repository now. Do you know why 3 checks were skipped and 
one is failing?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil opened a new pull request #9822: Add note in Updating.md about the change in `run_as_user` default



kaxil opened a new pull request #9822:
URL: https://github.com/apache/airflow/pull/9822


   Until Airflow 1.10.10 the default run_as_user config 
(https://airflow.readthedocs.io/en/1.10.10/configurations-ref.html#run-as-user) 
which defaulted it to root user `0` 
(https://github.com/apache/airflow/blob/96697180d79bfc90f6964a8e99f9dd441789177c/airflow/contrib/executors/kubernetes_executor.py#L295-L301)
   
   In Airflow 1.10.11 we changed it to `5`
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Target Github ISSUE in description if exists
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #9821: Fixes sqlalchemy flask failure with python 3.8.4



potiuk commented on pull request #9821:
URL: https://github.com/apache/airflow/pull/9821#issuecomment-658355716


   Yeah - somewhat version 5.0.1 became available for 3.6 but then removed for 
3.7. I think this is because I actually run two generate requirements in 
parallell (by mistake :) ). The 5.1.0  version of google ads is in 3.7+ but it 
appeared in 3.6 reqs.. I hope will be good now.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on pull request #9821: Fixes sqlalchemy flask failure with python 3.8.4



kaxil commented on pull request #9821:
URL: https://github.com/apache/airflow/pull/9821#issuecomment-658339979


   ```
 Could not find a version that satisfies the requirement google-ads==5.1.0 
(from -c /opt/airflow/requirements/requirements-python3.6.txt (line 139)) (from 
versions: 0.1.0, 0.2.0, 0.3.0, 0.4.0, 0.5.0, 0.5.1, 0.5.2, 0.6.0, 0.7.0, 1.0.0, 
1.0.1, 1.1.0, 1.1.1, 1.2.0, 1.3.0, 1.3.1, 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 
2.4.1, 3.0.0, 3.1.0, 3.1.1, 3.1.2, 3.2.0, 3.3.0, 4.0.0)
   No matching distribution found for google-ads==5.1.0 (from -c 
/opt/airflow/requirements/requirements-python3.6.txt (line 139))
   You are using pip version 19.0.2, however version 20.2b1 is available.
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch v1-10-test updated: Fixes sqlalchemy flask failure with python 3.8.4

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/v1-10-test by this push:
 new e983da7  Fixes sqlalchemy flask failure with python 3.8.4
e983da7 is described below

commit e983da7e9ef4f5a761e2ef75d5734a7580b83fd0
Author: Jarek Potiuk 
AuthorDate: Tue Jul 14 19:29:13 2020 +0200

Fixes sqlalchemy flask failure with python 3.8.4

(cherry picked from commit 021bb88bb82adf54b2299f723db0dc4d88fb6ab9)
---
 requirements/requirements-python2.7.txt | 134 +++-
 requirements/requirements-python3.5.txt |  87 +++--
 requirements/requirements-python3.6.txt |  53 +++--
 requirements/requirements-python3.7.txt |  52 ++---
 requirements/requirements-python3.8.txt |  51 ++--
 scripts/ci/libraries/_initialization.sh |  12 +--
 6 files changed, 190 insertions(+), 199 deletions(-)

diff --git a/requirements/requirements-python2.7.txt 
b/requirements/requirements-python2.7.txt
index 6973e5a..793e6b8 100644
--- a/requirements/requirements-python2.7.txt
+++ b/requirements/requirements-python2.7.txt
@@ -8,7 +8,7 @@ Flask-Caching==1.3.3
 Flask-JWT-Extended==3.24.1
 Flask-Login==0.4.1
 Flask-OpenID==1.2.5
-Flask-SQLAlchemy==2.4.3
+Flask-SQLAlchemy==2.4.4
 Flask-WTF==0.14.3
 Flask==1.1.2
 JPype1==0.7.1
@@ -22,10 +22,10 @@ PyJWT==1.7.1
 PyNaCl==1.4.0
 PySmbClient==0.1.5
 PyYAML==5.3.1
-Pygments==2.5.2
-SQLAlchemy-JSONField==0.8.0
+Pygments==2.6.1
+SQLAlchemy-JSONField==0.9.0
 SQLAlchemy==1.3.18
-Sphinx==1.8.5
+Sphinx==3.1.2
 Unidecode==1.1.1
 WTForms==2.3.1
 Werkzeug==0.16.1
@@ -36,44 +36,37 @@ amqp==2.6.0
 analytics-python==1.2.9
 ansiwrap==0.8.4
 apipkg==1.5
-apispec==2.0.2
+apispec==3.3.1
 appdirs==1.4.4
-argcomplete==1.11.1
+argcomplete==1.12.0
 asn1crypto==1.3.0
 aspy.yaml==1.3.0
-astroid==1.6.6
+astroid==2.4.2
 atlasclient==1.0.0
 atomicwrites==1.4.0
 attrs==19.3.0
 aws-sam-translator==1.25.0
 aws-xray-sdk==2.6.0
 azure-common==1.1.25
-azure-cosmos==3.1.2
+azure-cosmos==3.2.0
 azure-datalake-store==0.0.48
 azure-mgmt-containerinstance==1.5.0
-azure-mgmt-nspkg==3.0.2
-azure-mgmt-resource==10.0.0
+azure-mgmt-resource==10.1.0
 azure-nspkg==3.0.2
 azure-storage-blob==2.1.0
 azure-storage-common==2.1.0
-azure-storage-nspkg==3.1.0
 azure-storage==0.36.0
-backports-abc==0.5
-backports.functools-lru-cache==1.6.1
-backports.shutil-get-terminal-size==1.0.0
-backports.ssl-match-hostname==3.7.0.1
-backports.tempfile==1.0
-backports.weakref==1.0.post1
+backcall==0.2.0
 bcrypt==3.1.7
 beautifulsoup4==4.7.1
 billiard==3.6.3.0
 bleach==3.1.5
 blinker==1.4
-boto3==1.14.14
+boto3==1.14.20
 boto==2.49.0
-botocore==1.17.14
+botocore==1.17.20
 cached-property==1.5.1
-cachetools==3.1.1
+cachetools==4.1.1
 cassandra-driver==3.20.2
 cattrs==1.0.0
 celery==4.4.6
@@ -88,14 +81,11 @@ cloudant==0.5.10
 colorama==0.4.3
 colorlog==4.0.2
 configparser==3.5.3
-contextdecorator==0.10.0
-contextlib2==0.6.0.post1
-cookies==2.2.1
-coverage==5.1
+coverage==5.2
 croniter==0.3.34
 cryptography==2.9.2
-cx-Oracle==7.3.0
-datadog==0.37.1
+cx-Oracle==8.0.0
+datadog==0.38.0
 decorator==4.4.2
 defusedxml==0.6.0
 dill==0.3.2
@@ -110,27 +100,25 @@ elasticsearch-dsl==5.4.0
 elasticsearch==5.5.3
 email-validator==1.1.1
 entrypoints==0.3
-enum34==1.1.10
 execnet==1.7.1
-fastavro==0.23.5
+fastavro==0.23.6
 filelock==3.0.12
 flake8-colors==0.1.6
 flake8==3.8.3
-flaky==3.6.1
+flaky==3.7.0
 flask-swagger==0.2.13
-flower==0.9.4
+flower==0.9.5
 freezegun==0.3.15
+fsspec==0.7.4
 funcsigs==1.0.2
-functools32==3.2.3.post2
 future-fstrings==1.2.0
 future==0.18.2
-futures==3.3.0
-gcsfs==0.2.3
+gcsfs==0.6.2
 google-api-core==1.21.0
 google-api-python-client==1.9.3
-google-auth-httplib2==0.0.3
+google-auth-httplib2==0.0.4
 google-auth-oauthlib==0.4.1
-google-auth==1.18.0
+google-auth==1.19.0
 google-cloud-bigquery==1.25.0
 google-cloud-bigtable==1.2.1
 google-cloud-container==1.0.1
@@ -147,7 +135,7 @@ google-cloud-videointelligence==1.15.0
 google-cloud-vision==1.0.0
 google-resumable-media==0.5.1
 googleapis-common-protos==1.52.0
-graphviz==0.14
+graphviz==0.14.1
 grpc-google-iam-v1==0.12.3
 grpcio-gcp==0.2.2
 grpcio==1.30.0
@@ -155,22 +143,22 @@ gunicorn==19.10.0
 hdfs==2.5.8
 hmsclient==0.1.1
 httplib2==0.18.1
-humanize==0.5.1
+humanize==2.5.0
 hvac==0.10.4
-identify==1.4.21
+identify==1.4.23
 idna==2.10
 ijson==2.6.1
 imagesize==1.2.0
 importlib-metadata==1.7.0
 importlib-resources==3.0.0
-inflection==0.3.1
-ipaddress==1.0.23
+inflection==0.5.0
 ipdb==0.13.3
 ipython-genutils==0.2.0
-ipython==5.10.0
+ipython==7.9.0
 iso8601==0.1.12
 isodate==0.6.0
 itsdangerous==1.1.0
+jedi==0.17.1
 jira==2.0.0
 jmespath==0.10.0
 json-merge-patch==0.2
@@ -180,13 +168,12 @@ jsonpickle==1.4.1
 jsonpointer==2.0
 jsonschema==3.2.0
 junit-xml==1.9
-jupyter-client==5.3.5
+jupyter-client==6.1.6
 jupyter-core==4.6.3
-kombu==4.6.3
+kombu==4.6.11

[GitHub] [airflow] kaxil commented on pull request #9821: Fixes sqlalchemy flask failure with python 3.8.4



kaxil commented on pull request #9821:
URL: https://github.com/apache/airflow/pull/9821#issuecomment-658314755


   CI is failing :(



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #9806: Add redoc docs to sphinx



mik-laj commented on pull request #9806:
URL: https://github.com/apache/airflow/pull/9806#issuecomment-658311120


   @ad-m This will generate a file, but Sphinx won't be aware of it, so you 
won't be able to reference it in any way.
   ```
   /opt/airflowairflow/docs/index.rst:104: WARNING: toctree contains reference 
to nonexisting document 'redoc'
   ```
   This is a known problem for many plugins e.g. sphinx-autoapi also generates 
rst files on the local disk so that Sphinx can read them.
   
   How do I update this PS to use this plugin but it is not perfect. It doesn't 
allow you to easily specify the version of redoc library and copy JS code to 
output files.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk opened a new pull request #9821: Fixes sqlalchemy flask failure with python 3.8.4



potiuk opened a new pull request #9821:
URL: https://github.com/apache/airflow/pull/9821


   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Target Github ISSUE in description if exists
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #9747: [WIP] Add AzureBaseHook



mik-laj commented on pull request #9747:
URL: https://github.com/apache/airflow/pull/9747#issuecomment-658304397


   > I mean if the documentation is specific enough to tell you how to set the 
values for login password extra I personally think we shouldn't add custom 
fields.
   
   The extra fields and custom fields are the same. GCP adds its own fields, 
but this is only a visual matter. All information is. still in extra field, 
because they don't match any existing fields.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on pull request #9798: Guide for AI Platform (previously Machine Learning Engine) Operators



mik-laj commented on pull request #9798:
URL: https://github.com/apache/airflow/pull/9798#issuecomment-658300322


   Please look here: 
https://apache-airflow.slack.com/archives/CCPRP7943/p1594733324313800



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] vuppalli commented on pull request #9798: Guide for AI Platform (previously Machine Learning Engine) Operators



vuppalli commented on pull request #9798:
URL: https://github.com/apache/airflow/pull/9798#issuecomment-658297975


   @mik-laj Thank you so much for your comments! I have addressed them and 
pushed the changes. However, it looks like there are still some failed checks. 
Do you know what I can do to fix the issues?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (AIRFLOW-7014) Operator for Apache Kylin



[ 
https://issues.apache.org/jira/browse/AIRFLOW-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157493#comment-17157493
 ] 

ASF GitHub Bot commented on AIRFLOW-7014:
-

potiuk commented on pull request #9149:
URL: https://github.com/apache/airflow/pull/9149#issuecomment-658279014


   Merged. We had some temporary problem with Python 3.8.4 released last night 
breaking SQL Alchemy - that's why you got the errors. Thanks for your patience!



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Operator for Apache Kylin 
> --
>
> Key: AIRFLOW-7014
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7014
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: executors, operators
>Affects Versions: 1.10.0
>Reporter: Shao Feng Shi
>Assignee: liuyongheng
>Priority: Major
>
> Apache Kylin is an analytial data warehouse for big data. Kylin provides a 
> set of RESTful API for user to trigger the data loading, and also run SQL 
> queries against the OLAP cubes in sub-seconds latency. We developed a 
> KylinOperator within Airflow, so that user can easily trigger Kylin with 
> other tasks (hive, spark, etc), and plan to contribute into Airflow.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] potiuk commented on pull request #9149: [AIRFLOW-7014] Add Apache Kylin operator



potiuk commented on pull request #9149:
URL: https://github.com/apache/airflow/pull/9149#issuecomment-658279014


   Merged. We had some temporary problem with Python 3.8.4 released last night 
breaking SQL Alchemy - that's why you got the errors. Thanks for your patience!



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (AIRFLOW-7014) Operator for Apache Kylin



[ 
https://issues.apache.org/jira/browse/AIRFLOW-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157491#comment-17157491
 ] 

ASF GitHub Bot commented on AIRFLOW-7014:
-

potiuk merged pull request #9149:
URL: https://github.com/apache/airflow/pull/9149


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Operator for Apache Kylin 
> --
>
> Key: AIRFLOW-7014
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7014
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: executors, operators
>Affects Versions: 1.10.0
>Reporter: Shao Feng Shi
>Assignee: liuyongheng
>Priority: Major
>
> Apache Kylin is an analytial data warehouse for big data. Kylin provides a 
> set of RESTful API for user to trigger the data loading, and also run SQL 
> queries against the OLAP cubes in sub-seconds latency. We developed a 
> KylinOperator within Airflow, so that user can easily trigger Kylin with 
> other tasks (hive, spark, etc), and plan to contribute into Airflow.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-7014) Operator for Apache Kylin



[ 
https://issues.apache.org/jira/browse/AIRFLOW-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157492#comment-17157492
 ] 

ASF GitHub Bot commented on AIRFLOW-7014:
-

boring-cyborg[bot] commented on pull request #9149:
URL: https://github.com/apache/airflow/pull/9149#issuecomment-658278537


   Awesome work, congrats on your first merged pull request!
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Operator for Apache Kylin 
> --
>
> Key: AIRFLOW-7014
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7014
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: executors, operators
>Affects Versions: 1.10.0
>Reporter: Shao Feng Shi
>Assignee: liuyongheng
>Priority: Major
>
> Apache Kylin is an analytial data warehouse for big data. Kylin provides a 
> set of RESTful API for user to trigger the data loading, and also run SQL 
> queries against the OLAP cubes in sub-seconds latency. We developed a 
> KylinOperator within Airflow, so that user can easily trigger Kylin with 
> other tasks (hive, spark, etc), and plan to contribute into Airflow.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[airflow] branch master updated: Add kylin operator (#9149)

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/master by this push:
 new a2c5389  Add kylin operator (#9149)
a2c5389 is described below

commit a2c5389a60f68482a60eb40c67b1542d827c187e
Author: yongheng.liu <56812134+liuyonghengh...@users.noreply.github.com>
AuthorDate: Wed Jul 15 00:25:05 2020 +0800

Add kylin operator (#9149)

Co-authored-by: yongheng.liu 
---
 CONTRIBUTING.rst   |  16 +-
 INSTALL|  16 +-
 airflow/providers/apache/kylin/__init__.py |  16 ++
 .../apache/kylin/example_dags/__init__.py  |  16 ++
 .../apache/kylin/example_dags/example_kylin_dag.py | 136 +
 airflow/providers/apache/kylin/hooks/__init__.py   |  16 ++
 airflow/providers/apache/kylin/hooks/kylin.py  |  77 ++
 .../providers/apache/kylin/operators/__init__.py   |  16 ++
 .../providers/apache/kylin/operators/kylin_cube.py | 170 +
 airflow/utils/db.py|  11 ++
 docs/autoapi_templates/index.rst   |   4 +
 docs/operators-and-hooks-ref.rst   |   6 +
 requirements/requirements-python3.6.txt|   3 +-
 requirements/requirements-python3.7.txt|   3 +-
 requirements/requirements-python3.8.txt|   4 +-
 requirements/setup-3.6.md5 |   2 +-
 requirements/setup-3.7.md5 |   2 +-
 requirements/setup-3.8.md5 |   2 +-
 setup.py   |   7 +-
 tests/providers/apache/kylin/__init__.py   |  17 +++
 tests/providers/apache/kylin/hooks/__init__.py |  17 +++
 tests/providers/apache/kylin/hooks/test_kylin.py   |  60 
 tests/providers/apache/kylin/operators/__init__.py |  17 +++
 .../apache/kylin/operators/test_kylin_cube.py  | 170 +
 24 files changed, 780 insertions(+), 24 deletions(-)

diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
index 22a470c..e310efb 100644
--- a/CONTRIBUTING.rst
+++ b/CONTRIBUTING.rst
@@ -314,14 +314,14 @@ This is the full list of those extras:
   .. START EXTRAS HERE
 
 all_dbs, amazon, apache.atlas, apache_beam, apache.cassandra, apache.druid, 
apache.hdfs,
-apache.hive, apache.pinot, apache.webhdfs, async, atlas, aws, azure, 
cassandra, celery, cgroups,
-cloudant, cncf.kubernetes, dask, databricks, datadog, devel, devel_hadoop, 
doc, docker, druid,
-elasticsearch, exasol, facebook, gcp, gcp_api, github_enterprise, google, 
google_auth, grpc,
-hashicorp, hdfs, hive, jdbc, jira, kerberos, kubernetes, ldap, 
microsoft.azure, microsoft.mssql,
-microsoft.winrm, mongo, mssql, mysql, odbc, oracle, pagerduty, papermill, 
password, pinot, postgres,
-presto, qds, rabbitmq, redis, salesforce, samba, segment, sendgrid, sentry, 
singularity, slack,
-snowflake, spark, ssh, statsd, tableau, vertica, virtualenv, webhdfs, winrm, 
yandexcloud, all,
-devel_ci
+apache.hive, apache.kylin, apache.pinot, apache.webhdfs, async, atlas, aws, 
azure, cassandra,
+celery, cgroups, cloudant, cncf.kubernetes, dask, databricks, datadog, devel, 
devel_hadoop, doc,
+docker, druid, elasticsearch, exasol, facebook, gcp, gcp_api, 
github_enterprise, google,
+google_auth, grpc, hashicorp, hdfs, hive, jdbc, jira, kerberos, kubernetes, 
ldap, microsoft.azure,
+microsoft.mssql, microsoft.winrm, mongo, mssql, mysql, odbc, oracle, 
pagerduty, papermill, password,
+pinot, postgres, presto, qds, rabbitmq, redis, salesforce, samba, segment, 
sendgrid, sentry,
+singularity, slack, snowflake, spark, ssh, statsd, tableau, vertica, 
virtualenv, webhdfs, winrm,
+yandexcloud, all, devel_ci
 
   .. END EXTRAS HERE
 
diff --git a/INSTALL b/INSTALL
index fa4f672..64242c1 100644
--- a/INSTALL
+++ b/INSTALL
@@ -45,14 +45,14 @@ pip install . --constraint 
requirements/requirements-python3.7.txt
 # START EXTRAS HERE
 
 all_dbs, amazon, apache.atlas, apache_beam, apache.cassandra, apache.druid, 
apache.hdfs,
-apache.hive, apache.pinot, apache.webhdfs, async, atlas, aws, azure, 
cassandra, celery, cgroups,
-cloudant, cncf.kubernetes, dask, databricks, datadog, devel, devel_hadoop, 
doc, docker, druid,
-elasticsearch, exasol, facebook, gcp, gcp_api, github_enterprise, google, 
google_auth, grpc,
-hashicorp, hdfs, hive, jdbc, jira, kerberos, kubernetes, ldap, 
microsoft.azure, microsoft.mssql,
-microsoft.winrm, mongo, mssql, mysql, odbc, oracle, pagerduty, papermill, 
password, pinot, postgres,
-presto, qds, rabbitmq, redis, salesforce, samba, segment, sendgrid, sentry, 
singularity, slack,
-snowflake, spark, ssh, statsd, tableau, vertica, virtualenv, webhdfs, winrm, 
yandexcloud, all,
-devel_ci
+apache.hive, apache.kylin, apache.pinot, apache.webhdfs, async, atlas, aws, 
azure, cassandra,
+celery, cgroups, cloudant,

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #9149: [AIRFLOW-7014] Add Apache Kylin operator



boring-cyborg[bot] commented on pull request #9149:
URL: https://github.com/apache/airflow/pull/9149#issuecomment-658278537


   Awesome work, congrats on your first merged pull request!
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk merged pull request #9149: [AIRFLOW-7014] Add Apache Kylin operator



potiuk merged pull request #9149:
URL: https://github.com/apache/airflow/pull/9149


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #9667: Allow `replace` flag in gcs_to_gcs operator.



potiuk commented on pull request #9667:
URL: https://github.com/apache/airflow/pull/9667#issuecomment-658276991


   Thanks @royberkoweee :). Sometimes we miss a PR. Gentle Ping as you did is 
helpful!. Thanks!



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #9667: Allow `replace` flag in gcs_to_gcs operator.



boring-cyborg[bot] commented on pull request #9667:
URL: https://github.com/apache/airflow/pull/9667#issuecomment-658276627


   Awesome work, congrats on your first merged pull request!
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated (4636fc6 -> ed5004c)

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git.


from 4636fc6  Python 3.8.4 release breaks our builds (#9820)
 add ed5004c  Allow `replace` flag in gcs_to_gcs operator. (#9667)

No new revisions were added by this update.

Summary of changes:
 .../providers/google/cloud/transfers/gcs_to_gcs.py| 19 +++
 .../google/cloud/transfers/test_gcs_to_gcs.py | 15 +++
 2 files changed, 34 insertions(+)

[GitHub] [airflow] potiuk merged pull request #9667: Allow `replace` flag in gcs_to_gcs operator.



potiuk merged pull request #9667:
URL: https://github.com/apache/airflow/pull/9667


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #9667: Allow `replace` flag in gcs_to_gcs operator.



potiuk commented on pull request #9667:
URL: https://github.com/apache/airflow/pull/9667#issuecomment-658276362


   Ah missed it :)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated: Python 3.8.4 release breaks our builds (#9820)

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/master by this push:
 new 4636fc6  Python 3.8.4 release breaks our builds (#9820)
4636fc6 is described below

commit 4636fc6ededbf604daa175a7cd9cff850aba40bb
Author: Jarek Potiuk 
AuthorDate: Tue Jul 14 18:15:12 2020 +0200

Python 3.8.4 release breaks our builds (#9820)
---
 breeze  |  2 +-
 scripts/ci/libraries/_build_images.sh   |  8 
 scripts/ci/libraries/_initialization.sh | 12 
 3 files changed, 17 insertions(+), 5 deletions(-)

diff --git a/breeze b/breeze
index c2a6edc..2b34b17 100755
--- a/breeze
+++ b/breeze
@@ -526,7 +526,7 @@ function prepare_command_files() {
 export COMPOSE_PROD_FILE
 
 # Base python image for the build
-export PYTHON_BASE_IMAGE=python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster
+export PYTHON_BASE_IMAGE=python:${PYTHON_BASE_IMAGE_VERSION}-slim-buster
 export 
AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci"
 export 
AIRFLOW_PROD_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}"
 export 
BUILT_IMAGE_FLAG_FILE="${BUILD_CACHE_DIR}/${BRANCH_NAME}/.built_${PYTHON_MAJOR_MINOR_VERSION}"
diff --git a/scripts/ci/libraries/_build_images.sh 
b/scripts/ci/libraries/_build_images.sh
index 26ab1c4..ed894a1 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -334,14 +334,14 @@ function prepare_ci_build() {
 fi
 export 
CACHE_IMAGE_PREFIX=${CACHE_IMAGE_PREFX:=${GITHUB_ORGANISATION}/${GITHUB_REPO}}
 export 
CACHED_AIRFLOW_CI_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_CI_BASE_TAG}"
-export 
CACHED_PYTHON_BASE_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
+export 
CACHED_PYTHON_BASE_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/python:${PYTHON_BASE_IMAGE_VERSION}-slim-buster"
 else
 export CACHED_AIRFLOW_CI_IMAGE=""
 export CACHED_PYTHON_BASE_IMAGE=""
 fi
 export 
AIRFLOW_BUILD_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}/${AIRFLOW_CI_BASE_TAG}"
 export 
AIRFLOW_CI_IMAGE_DEFAULT="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}-ci"
-export PYTHON_BASE_IMAGE="python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
+export PYTHON_BASE_IMAGE="python:${PYTHON_BASE_IMAGE_VERSION}-slim-buster"
 export 
BUILT_IMAGE_FLAG_FILE="${BUILD_CACHE_DIR}/${BRANCH_NAME}/.built_${PYTHON_MAJOR_MINOR_VERSION}"
 if [[ "${DEFAULT_PYTHON_MAJOR_MINOR_VERSION}" == 
"${PYTHON_MAJOR_MINOR_VERSION}" ]]; then
 export DEFAULT_IMAGE="${AIRFLOW_CI_IMAGE_DEFAULT}"
@@ -621,7 +621,7 @@ function prepare_prod_build() {
 export 
AIRFLOW_PROD_BUILD_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_PROD_BASE_TAG}-build"
 export 
AIRFLOW_PROD_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_PROD_BASE_TAG}"
 export 
AIRFLOW_PROD_IMAGE_DEFAULT="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}"
-export PYTHON_BASE_IMAGE="python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
+export PYTHON_BASE_IMAGE="python:${PYTHON_BASE_IMAGE_VERSION}-slim-buster"
 if [[ "${DEFAULT_PYTHON_MAJOR_MINOR_VERSION}" == 
"${PYTHON_MAJOR_MINOR_VERSION}" ]]; then
 export DEFAULT_IMAGE="${AIRFLOW_PROD_IMAGE_DEFAULT}"
 else
@@ -646,7 +646,7 @@ function prepare_prod_build() {
 export 
CACHE_IMAGE_PREFIX=${CACHE_IMAGE_PREFX:=${GITHUB_ORGANISATION}/${GITHUB_REPO}}
 export 
CACHED_AIRFLOW_PROD_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_PROD_BASE_TAG}"
 export 
CACHED_AIRFLOW_PROD_BUILD_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_PROD_BASE_TAG}-build"
-export 
CACHED_PYTHON_BASE_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
+export 
CACHED_PYTHON_BASE_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/python:${PYTHON_BASE_IMAGE_VERSION}-slim-buster"
 else
 export CACHED_AIRFLOW_PROD_IMAGE=""
 export CACHED_AIRFLOW_PROD_BUILD_IMAGE=""
diff --git a/scripts/ci/libraries/_initialization.sh 
b/scripts/ci/libraries/_initialization.sh
index f7ed5a3..6e9c5c3 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -21,6 +21,18 @@ function initialize_common_environment {
 # default python Major/Minor version
 PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:="3.6"}
 
+PYTHON_BASE_IMAGE_VERSION=${PYTHON_MAJOR_MINOR_VERSION}
+if [[ ${PYTHON_BASE_IMAGE_VERSION} == "3.8" ]]; then
+# Temporary fix for Python 3.8.4 failing our tests
+# TODO: Remove after it gets fixed
+PYTHON_BASE_IMAGE_VERSION="3.8.3"
+echo

[GitHub] [airflow] potiuk merged pull request #9820: Python 3.8.4 release breaks our builds



potiuk merged pull request #9820:
URL: https://github.com/apache/airflow/pull/9820


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch v1-10-test updated: Python 3.8.4 release breaks our builds

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/v1-10-test by this push:
 new 699382b  Python 3.8.4 release breaks our builds
699382b is described below

commit 699382b603debb6bf0a1dbed78580dd9564148b1
Author: Jarek Potiuk 
AuthorDate: Tue Jul 14 16:17:51 2020 +0200

Python 3.8.4 release breaks our builds
---
 breeze  |  2 +-
 scripts/ci/libraries/_build_images.sh   |  8 
 scripts/ci/libraries/_initialization.sh | 12 
 3 files changed, 17 insertions(+), 5 deletions(-)

diff --git a/breeze b/breeze
index 01927e5..3e8d2c3 100755
--- a/breeze
+++ b/breeze
@@ -509,7 +509,7 @@ function prepare_command_files() {
 export COMPOSE_PROD_FILE
 
 # Base python image for the build
-export PYTHON_BASE_IMAGE=python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster
+export PYTHON_BASE_IMAGE=python:${PYTHON_BASE_IMAGE_VERSION}-slim-buster
 export 
AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci"
 export 
AIRFLOW_PROD_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}"
 export 
BUILT_IMAGE_FLAG_FILE="${BUILD_CACHE_DIR}/${BRANCH_NAME}/.built_${PYTHON_MAJOR_MINOR_VERSION}"
diff --git a/scripts/ci/libraries/_build_images.sh 
b/scripts/ci/libraries/_build_images.sh
index 26ab1c4..ed894a1 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -334,14 +334,14 @@ function prepare_ci_build() {
 fi
 export 
CACHE_IMAGE_PREFIX=${CACHE_IMAGE_PREFX:=${GITHUB_ORGANISATION}/${GITHUB_REPO}}
 export 
CACHED_AIRFLOW_CI_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_CI_BASE_TAG}"
-export 
CACHED_PYTHON_BASE_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
+export 
CACHED_PYTHON_BASE_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/python:${PYTHON_BASE_IMAGE_VERSION}-slim-buster"
 else
 export CACHED_AIRFLOW_CI_IMAGE=""
 export CACHED_PYTHON_BASE_IMAGE=""
 fi
 export 
AIRFLOW_BUILD_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}/${AIRFLOW_CI_BASE_TAG}"
 export 
AIRFLOW_CI_IMAGE_DEFAULT="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}-ci"
-export PYTHON_BASE_IMAGE="python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
+export PYTHON_BASE_IMAGE="python:${PYTHON_BASE_IMAGE_VERSION}-slim-buster"
 export 
BUILT_IMAGE_FLAG_FILE="${BUILD_CACHE_DIR}/${BRANCH_NAME}/.built_${PYTHON_MAJOR_MINOR_VERSION}"
 if [[ "${DEFAULT_PYTHON_MAJOR_MINOR_VERSION}" == 
"${PYTHON_MAJOR_MINOR_VERSION}" ]]; then
 export DEFAULT_IMAGE="${AIRFLOW_CI_IMAGE_DEFAULT}"
@@ -621,7 +621,7 @@ function prepare_prod_build() {
 export 
AIRFLOW_PROD_BUILD_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_PROD_BASE_TAG}-build"
 export 
AIRFLOW_PROD_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${AIRFLOW_PROD_BASE_TAG}"
 export 
AIRFLOW_PROD_IMAGE_DEFAULT="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}"
-export PYTHON_BASE_IMAGE="python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
+export PYTHON_BASE_IMAGE="python:${PYTHON_BASE_IMAGE_VERSION}-slim-buster"
 if [[ "${DEFAULT_PYTHON_MAJOR_MINOR_VERSION}" == 
"${PYTHON_MAJOR_MINOR_VERSION}" ]]; then
 export DEFAULT_IMAGE="${AIRFLOW_PROD_IMAGE_DEFAULT}"
 else
@@ -646,7 +646,7 @@ function prepare_prod_build() {
 export 
CACHE_IMAGE_PREFIX=${CACHE_IMAGE_PREFX:=${GITHUB_ORGANISATION}/${GITHUB_REPO}}
 export 
CACHED_AIRFLOW_PROD_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_PROD_BASE_TAG}"
 export 
CACHED_AIRFLOW_PROD_BUILD_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/${AIRFLOW_PROD_BASE_TAG}-build"
-export 
CACHED_PYTHON_BASE_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/python:${PYTHON_MAJOR_MINOR_VERSION}-slim-buster"
+export 
CACHED_PYTHON_BASE_IMAGE="${CACHE_REGISTRY}/${CACHE_IMAGE_PREFIX}/python:${PYTHON_BASE_IMAGE_VERSION}-slim-buster"
 else
 export CACHED_AIRFLOW_PROD_IMAGE=""
 export CACHED_AIRFLOW_PROD_BUILD_IMAGE=""
diff --git a/scripts/ci/libraries/_initialization.sh 
b/scripts/ci/libraries/_initialization.sh
index 5f2a742..1a7e398 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -21,6 +21,18 @@ function initialize_common_environment {
 # default python Major/Minor version
 PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:="3.6"}
 
+PYTHON_BASE_IMAGE_VERSION=${PYTHON_MAJOR_MINOR_VERSION}
+if [[ ${PYTHON_BASE_IMAGE_VERSION} == "3.8" ]]; then
+# Temporary fix for Python 3.8.4 failing our tests
+# TODO: Remove after it gets fixed
+PYTHON_BASE_IMAGE_VERSION="3.8.3"
+echo

[GitHub] [airflow] houqp commented on a change in pull request #9631: Add function to get current context



houqp commented on a change in pull request #9631:
URL: https://github.com/apache/airflow/pull/9631#discussion_r454433163



##
File path: airflow/models/taskinstance.py
##
@@ -1132,7 +1133,8 @@ def signal_handler(signum, frame):  # pylint: 
disable=unused-argument
 self._run_execute_callback(context, task)
 
 # Execute the task
-result = self._execute_task(context, task_copy)
+with set_current_context(context):

Review comment:
   If we can move the logic into functional dag module, that would be 
definitely more explicit and easier to maintain going forward.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated: For now cloud tools are not needed in CI (#9818)

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/master by this push:
 new 69f82e6  For now cloud tools are not needed in CI (#9818)
69f82e6 is described below

commit 69f82e66af54fb85a07ee6c7c85b8d4f5140e758
Author: Jarek Potiuk 
AuthorDate: Tue Jul 14 16:35:33 2020 +0200

For now cloud tools are not needed in CI (#9818)

Currently there is "unbound" variable error printed in CI logs
because of that.
---
 scripts/ci/in_container/entrypoint_ci.sh | 8 +---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/scripts/ci/in_container/entrypoint_ci.sh 
b/scripts/ci/in_container/entrypoint_ci.sh
index b8302b3..6ff9134 100755
--- a/scripts/ci/in_container/entrypoint_ci.sh
+++ b/scripts/ci/in_container/entrypoint_ci.sh
@@ -45,9 +45,11 @@ RUN_TESTS=${RUN_TESTS:="false"}
 CI=${CI:="false"}
 INSTALL_AIRFLOW_VERSION="${INSTALL_AIRFLOW_VERSION:=""}"
 
-# Create links for useful CLI tools
-# shellcheck source=scripts/ci/run_cli_tool.sh
-source <(bash scripts/ci/run_cli_tool.sh)
+if [[ ${CI} == "false" ]]; then
+# Create links for useful CLI tools
+# shellcheck source=scripts/ci/run_cli_tool.sh
+source <(bash scripts/ci/run_cli_tool.sh)
+fi
 
 if [[ ${AIRFLOW_VERSION} == *1.10* || ${INSTALL_AIRFLOW_VERSION} == *1.10* ]]; 
then
 export RUN_AIRFLOW_1_10="true"

[GitHub] [airflow] potiuk merged pull request #9818: For now cloud tools are not needed in CI



potiuk merged pull request #9818:
URL: https://github.com/apache/airflow/pull/9818


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated: Remove package.json and yarn.lock from the prod image (#9814)

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/master by this push:
 new 593a0dd  Remove package.json and yarn.lock from the prod image (#9814)
593a0dd is described below

commit 593a0ddaae2deaa283c260a32187cf3c27ec3e7d
Author: Jarek Potiuk 
AuthorDate: Tue Jul 14 16:34:21 2020 +0200

Remove package.json and yarn.lock from the prod image (#9814)

Closes #9810
---
 Dockerfile | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Dockerfile b/Dockerfile
index 4d182e1..55cbe2f5 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -224,6 +224,7 @@ RUN 
AIRFLOW_SITE_PACKAGE="/root/.local/lib/python${PYTHON_MAJOR_MINOR_VERSION}/s
 yarn --cwd "${WWW_DIR}" install --frozen-lockfile --no-cache; \
 yarn --cwd "${WWW_DIR}" run prod; \
 rm -rf "${WWW_DIR}/node_modules"; \
+rm -vf 
"${WWW_DIR}"/{package.json,yarn.lock,.eslintignore,.eslintrc,.stylelintignore,.stylelintrc,compile_assets.sh,webpack.config.js}
 ;\
 fi
 
 # make sure that all directories and files in .local are also group accessible

[GitHub] [airflow] potiuk closed issue #9810: Yarn.lock should not be present in the prod image



potiuk closed issue #9810:
URL: https://github.com/apache/airflow/issues/9810


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk merged pull request #9814: Remove package.json and yarn.lock from the prod image



potiuk merged pull request #9814:
URL: https://github.com/apache/airflow/pull/9814


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on a change in pull request #9820: Python 3.8.4 release breaks our builds



potiuk commented on a change in pull request #9820:
URL: https://github.com/apache/airflow/pull/9820#discussion_r454401506



##
File path: scripts/ci/libraries/_initialization.sh
##
@@ -21,6 +21,13 @@ function initialize_common_environment {
 # default python Major/Minor version
 PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:="3.6"}
 
+PYTHON_BASE_IMAGE_VERSION=${PYTHON_MAJOR_MINOR_VERSION}
+if [[ ${PYTHON_BASE_IMAGE_VERSION} == "3.8" ]]; then
+# Temporary fix for Python 3.8.4 failing our tests
+# TODO: Remove after it gets fixed
+PYTHON_BASE_IMAGE_VERSION="3.8.3"

Review comment:
   Good idea. Added.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] feluelle commented on a change in pull request #9820: Python 3.8.4 release breaks our builds



feluelle commented on a change in pull request #9820:
URL: https://github.com/apache/airflow/pull/9820#discussion_r454398222



##
File path: scripts/ci/libraries/_initialization.sh
##
@@ -21,6 +21,13 @@ function initialize_common_environment {
 # default python Major/Minor version
 PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:="3.6"}
 
+PYTHON_BASE_IMAGE_VERSION=${PYTHON_MAJOR_MINOR_VERSION}
+if [[ ${PYTHON_BASE_IMAGE_VERSION} == "3.8" ]]; then
+# Temporary fix for Python 3.8.4 failing our tests
+# TODO: Remove after it gets fixed
+PYTHON_BASE_IMAGE_VERSION="3.8.3"

Review comment:
   Can we `echo` this?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #9820: Python 3.8.4 release breaks our builds



potiuk commented on pull request #9820:
URL: https://github.com/apache/airflow/pull/9820#issuecomment-658211367


   > Can you also add it to the README 
https://github.com/apache/airflow#additional-notes-on-python-version-requirements
 ? :)
   
   I will - but I think we need to know the root cause before. It might be 
something that will be fixed in sqlalchemy - in which case it might work fine 
with 3.8.4 when they fix it. 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] jhtimmins commented on issue #8111: Authentication in API



jhtimmins commented on issue #8111:
URL: https://github.com/apache/airflow/issues/8111#issuecomment-658210379


   @mik-laj have you had a chance to look at what I added in 
https://github.com/jhtimmins/scratch-airflow-auth/blob/master/auth/__init__.py? 
Really that code is about defining the interface for interacting with the 
authentication system. Two questions:
   1. What do you think of the interface?
   2. Should I implement any additional pieces of the auth system for this 
specific issue?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk opened a new pull request #9820: Python 3.8.4 release breaks our builds



potiuk opened a new pull request #9820:
URL: https://github.com/apache/airflow/pull/9820


   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Target Github ISSUE in description if exists
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (AIRFLOW-3363) Plugin templates not rendered by Flask appbuilder's baseview

2020-07-14 Thread igor (Jira)



[ 
https://issues.apache.org/jira/browse/AIRFLOW-3363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157404#comment-17157404
 ] 

igor commented on AIRFLOW-3363:
---

Hello [~ran.zvi]

I had the same issue, but after investigation it was noticed that template path 
in Blueprints was incorrect. Please do remember that you need to create and 
register Blueprint for your custom view.

> Plugin templates not rendered by Flask appbuilder's baseview
> 
>
> Key: AIRFLOW-3363
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3363
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: plugins
>Affects Versions: 1.10.0
> Environment: Docker 
>Reporter: Ran Zvi
>Priority: Major
>
> Hello, we're having issues with the new F.A.B plugins (rbac), the 
> documentation lacks the import for *{{AppBuilderBaseView}}*{{:}}
>  
> {code:java}
> # Creating a flask appbuilder BaseView
>  class TestAppBuilderBaseView(AppBuilderBaseView):
>  @expose("/")
>  def test(self):
>  return self.render("test_plugin/test.html", content="Hello galaxy!"){code}
>  However after a bit of digging I found out it is probably imported from F.A.B
> {code:java}
> from flask_appbuilder import BaseView as AppBuilderBaseView{code}
>  
> The next issue is that the class lacks a *{{render}}* function and F.A.B only 
> provides a *{{render_template}}* function which ignores the 
> *{{plugins_folder}}*  definition in *{{airflow.cfg}}* and the Blueprint's 
> *{{template_folder}}* definition in the plugin's code.
> We've found out that in F.A.B's *{{Environment}}* Object created by Airflow 
> the only location defined for template location is under *{{www_rbac}}*{{.}}
> We're unable to make any progress due to this Bug, does anyone know how to  
> Thanks in advance!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] royberkoweee commented on pull request #9667: Allow `replace` flag in gcs_to_gcs operator.



royberkoweee commented on pull request #9667:
URL: https://github.com/apache/airflow/pull/9667#issuecomment-658197844


   Hey guys, how does the process look like from this point? Merging into 
master etc 



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] branch master updated: Add type annotations for redis provider (#9815)

2020-07-14 Thread kamilbregula

This is an automated email from the ASF dual-hosted git repository.

kamilbregula pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/master by this push:
 new 0a2acf0  Add type annotations for redis provider (#9815)
0a2acf0 is described below

commit 0a2acf0b6542b717f87dee6bbff43397bbb0e83b
Author: Alexander Sutcliffe <41974784+scrambldchan...@users.noreply.github.com>
AuthorDate: Tue Jul 14 15:52:38 2020 +0200

Add type annotations for redis provider (#9815)
---
 airflow/providers/redis/hooks/redis.py |  2 +-
 airflow/providers/redis/operators/redis_publish.py | 12 +++-
 airflow/providers/redis/sensors/redis_key.py   |  6 --
 airflow/providers/redis/sensors/redis_pub_sub.py   |  6 --
 4 files changed, 16 insertions(+), 10 deletions(-)

diff --git a/airflow/providers/redis/hooks/redis.py 
b/airflow/providers/redis/hooks/redis.py
index b83959a..52431f5 100644
--- a/airflow/providers/redis/hooks/redis.py
+++ b/airflow/providers/redis/hooks/redis.py
@@ -33,7 +33,7 @@ class RedisHook(BaseHook):
 Also you can set ssl parameters as:
 ``{"ssl": true, "ssl_cert_reqs": "require", "ssl_cert_file": 
"/path/to/cert.pem", etc}``.
 """
-def __init__(self, redis_conn_id='redis_default'):
+def __init__(self, redis_conn_id: str = 'redis_default') -> None:
 """
 Prepares hook to connect to a Redis database.
 
diff --git a/airflow/providers/redis/operators/redis_publish.py 
b/airflow/providers/redis/operators/redis_publish.py
index 6734b42..8c357af 100644
--- a/airflow/providers/redis/operators/redis_publish.py
+++ b/airflow/providers/redis/operators/redis_publish.py
@@ -16,6 +16,8 @@
 # specific language governing permissions and limitations
 # under the License.
 
+from typing import Dict
+
 from airflow.models import BaseOperator
 from airflow.providers.redis.hooks.redis import RedisHook
 from airflow.utils.decorators import apply_defaults
@@ -38,17 +40,17 @@ class RedisPublishOperator(BaseOperator):
 @apply_defaults
 def __init__(
 self,
-channel,
-message,
-redis_conn_id='redis_default',
-*args, **kwargs):
+channel: str,
+message: str,
+redis_conn_id: str = 'redis_default',
+*args, **kwargs) -> None:
 
 super().__init__(*args, **kwargs)
 self.redis_conn_id = redis_conn_id
 self.channel = channel
 self.message = message
 
-def execute(self, context):
+def execute(self, context: Dict) -> None:
 """
 Publish the message to Redis channel
 
diff --git a/airflow/providers/redis/sensors/redis_key.py 
b/airflow/providers/redis/sensors/redis_key.py
index e53be94..59167e9 100644
--- a/airflow/providers/redis/sensors/redis_key.py
+++ b/airflow/providers/redis/sensors/redis_key.py
@@ -15,6 +15,8 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+from typing import Dict
+
 from airflow.providers.redis.hooks.redis import RedisHook
 from airflow.sensors.base_sensor_operator import BaseSensorOperator
 from airflow.utils.decorators import apply_defaults
@@ -28,11 +30,11 @@ class RedisKeySensor(BaseSensorOperator):
 ui_color = '#f0eee4'
 
 @apply_defaults
-def __init__(self, key, redis_conn_id, *args, **kwargs):
+def __init__(self, key: str, redis_conn_id: str, *args, **kwargs) -> None:
 super().__init__(*args, **kwargs)
 self.redis_conn_id = redis_conn_id
 self.key = key
 
-def poke(self, context):
+def poke(self, context: Dict) -> bool:
 self.log.info('Sensor checks for existence of key: %s', self.key)
 return RedisHook(self.redis_conn_id).get_conn().exists(self.key)
diff --git a/airflow/providers/redis/sensors/redis_pub_sub.py 
b/airflow/providers/redis/sensors/redis_pub_sub.py
index 1e3fecb..9be2b56 100644
--- a/airflow/providers/redis/sensors/redis_pub_sub.py
+++ b/airflow/providers/redis/sensors/redis_pub_sub.py
@@ -16,6 +16,8 @@
 # specific language governing permissions and limitations
 # under the License.
 
+from typing import Dict, List, Union
+
 from airflow.providers.redis.hooks.redis import RedisHook
 from airflow.sensors.base_sensor_operator import BaseSensorOperator
 from airflow.utils.decorators import apply_defaults
@@ -34,14 +36,14 @@ class RedisPubSubSensor(BaseSensorOperator):
 ui_color = '#f0eee4'
 
 @apply_defaults
-def __init__(self, channels, redis_conn_id, *args, **kwargs):
+def __init__(self, channels: Union[List[str], str], redis_conn_id: str, 
*args, **kwargs) -> None:
 super().__init__(*args, **kwargs)
 self.channels = channels
 self.redis_conn_id = redis_conn_id
 self.pubsub = 
RedisHook(redis_conn_id=self.redis_conn_id).get_conn().pubsub()

[GitHub] [airflow] mik-laj merged pull request #9815: Adding type annotations to redis provider



mik-laj merged pull request #9815:
URL: https://github.com/apache/airflow/pull/9815


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #9815: Adding type annotations to redis provider



boring-cyborg[bot] commented on pull request #9815:
URL: https://github.com/apache/airflow/pull/9815#issuecomment-658192770


   Awesome work, congrats on your first merged pull request!
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] morrme commented on a change in pull request #9775: improve typing for datadog provider



morrme commented on a change in pull request #9775:
URL: https://github.com/apache/airflow/pull/9775#discussion_r454371731



##
File path: airflow/providers/datadog/hooks/datadog.py
##
@@ -115,8 +120,16 @@ def query_metric(self,
 return response
 
 # pylint: disable=too-many-arguments
-def post_event(self, title, text, aggregation_key=None, alert_type=None, 
date_happened=None,
-   handle=None, priority=None, related_event_id=None, 
tags=None, device_name=None):
+def post_event(self, title: str,
+   text: str,
+   aggregation_key: Optional[str] = None,
+   alert_type: Optional[str] = None,
+   date_happened: Optional[int] = None,
+   handle: Optional[str] = None,
+   priority: Optional[str] = None,
+   related_event_id: Optional[str] = None,

Review comment:
   > `related_event_id` is of type `int64` in the [Datadog 
API](https://docs.datadoghq.com/api/v1/events/#get-an-event). So in my opinion 
an `int` would be the proper choice here. What do you think?
   
   I agree, but since the file referenced another type, i wanted to be sure. 
Thank you!





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on issue #8111: Authentication in API



mik-laj commented on issue #8111:
URL: https://github.com/apache/airflow/issues/8111#issuecomment-658191051


   @ad-m  I do not want to introduce any requirements on the client API. If you 
want, you can use OAuth2, but you can also implement Kerberos-based or other 
fashionable protocol.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] feluelle commented on a change in pull request #9819: Import Iterable from collections.abc in firestore operators



feluelle commented on a change in pull request #9819:
URL: https://github.com/apache/airflow/pull/9819#discussion_r454367658



##
File path: airflow/providers/google/firebase/operators/firestore.py
##
@@ -15,7 +15,7 @@
 # specific language governing permissions and limitations
 # under the License.
 
-from collections import Iterable
+from collections.abc import Iterable

Review comment:
   True.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[airflow] 01/02: Fix small errors in image building documentation (#9792)

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 2f113fb688274dfc374b785b3771cc7fb9f2bf10
Author: Jarek Potiuk 
AuthorDate: Mon Jul 13 19:23:55 2020 +0200

Fix small errors in image building documentation (#9792)

(cherry picked from commit 56f7a2b6e6955f2233d603886266a8d8b286a29b)
---
 BREEZE.rst |  9 -
 IMAGES.rst | 10 ++
 2 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/BREEZE.rst b/BREEZE.rst
index dbd8b1e..dfe3c11 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -562,7 +562,7 @@ This install additional pypi dependency - torchio in 
specified version.
 
 .. code-block:: bash
 
- ./breeze build-image --production-image --additional-dev-deps 
"libasound2" \
+ ./breeze build-image --production-image --additional-dev-deps 
"libasound2-dev" \
 --additional-runtime-deps "libasound2"
 
 This install additional apt dependencies - ``libasound2-dev`` in build image 
and ``libasound`` in the
@@ -573,6 +573,13 @@ as an intermediate step to build the final image. Usually 
names of the ``dev`` d
 suffix and they need to also be paired with corresponding runtime dependency 
added for the runtime image
 (without -dev).
 
+.. code-block:: bash
+
+ ./breeze build-image --production-image --python 3.7 
--additional-dev-deps "libasound2-dev" \
+--additional-runtime-deps "libasound2"
+
+Same as above but uses python 3.7.
+
 .. raw:: html
 
 
diff --git a/IMAGES.rst b/IMAGES.rst
index 5c722b7..b1890e1 100644
--- a/IMAGES.rst
+++ b/IMAGES.rst
@@ -555,10 +555,9 @@ additional apt dev and runtime dependencies.
 --build-arg PYTHON_MAJOR_MINOR_VERSION=3.7 \
 --build-arg AIRFLOW_INSTALL_SOURCES="apache-airflow" \
 --build-arg AIRFLOW_INSTALL_VERSION="==1.10.10" \
---build-arg 
CONSTRAINT_REQUIREMENTS="https://raw.githubusercontent.com/apache/airflow/1.10.10/requirements/requirements-python3.7.txt;
 \
---build-arg 
ENTRYPOINT_FILE="https://raw.githubusercontent.com/apache/airflow/1.10.10/entrypoint.sh;
 \
---build-arg AIRFLOW_SOURCES_FROM="entrypoint.sh" \
---build-arg AIRFLOW_SOURCES_TO="/entrypoint" \
+--build-arg 
CONSTRAINT_REQUIREMENTS="https://raw.githubusercontent.com/apache/airflow/1.10.11/requirements/requirements-python3.7.txt;
 \
+--build-arg AIRFLOW_SOURCES_FROM="empty" \
+--build-arg AIRFLOW_SOURCES_TO="/empty" \
 --build-arg ADDITIONAL_AIRFLOW_EXTRAS="jdbc"
 --build-arg ADDITIONAL_DEV_DEPS="gcc g++"
 --build-arg ADDITIONAL_RUNTIME_DEPS="default-jre-headless"
@@ -647,6 +646,9 @@ The entrypoint performs those operations:
 Using the PROD image
 
 
+The entrypoint in the PROD image contains all the initialisation needed for 
tests to be immediately executed.
+It is copied from ``scripts/ci/in_container/entrypoint_prod.sh``.
+
 The PROD image entrypoint works as follows:
 
 * In case the user is not "airflow" (with undefined user id) and the group id 
of the user is set to 0 (root),

[airflow] branch v1-10-test updated (e434014 -> 90b66d0)

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a change to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git.


omit e434014  Update README to add Py 3.8 in supported versions (#9804)
omit 702ad72  Fix small errors in image building documentation (#9792)
omit 8af6277  Enable pretty output in mypy (#9785)
 new 2f113fb  Fix small errors in image building documentation (#9792)
 new 90b66d0  Update README to add Py 3.8 in supported versions (#9804)

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (e434014)
\
 N -- N -- N   refs/heads/v1-10-test (90b66d0)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 setup.cfg | 3 ---
 1 file changed, 3 deletions(-)

[airflow] 02/02: Update README to add Py 3.8 in supported versions (#9804)

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git

commit 90b66d0333bbebd36a1572c04a14fade6afe2f3d
Author: Kaxil Naik 
AuthorDate: Tue Jul 14 08:11:54 2020 +0100

Update README to add Py 3.8 in supported versions (#9804)

(cherry picked from commit 468e9507adb9b31d858f7f3c78ece1ad62b1a789)
---
 README.md | 1 -
 1 file changed, 1 deletion(-)

diff --git a/README.md b/README.md
index 96ce05f..940e972 100644
--- a/README.md
+++ b/README.md
@@ -79,7 +79,6 @@ Apache Airflow is tested with:
 ### Additional notes on Python version requirements
 
 * Stable version [requires](https://github.com/apache/airflow/issues/8162) at 
least Python 3.5.3 when using Python 3
-* Stable version is currently incompatible with Python 3.8 due to [a known 
compatibility issue](https://github.com/Tinche/cattrs/issues/77) with a 
dependent library
 
 ## Getting started
 Please visit the Airflow Platform documentation (latest **stable** release) 
for help with [installing 
Airflow](https://airflow.apache.org/installation.html), getting a [quick 
start](https://airflow.apache.org/start.html), or a more complete 
[tutorial](https://airflow.apache.org/tutorial.html).

[GitHub] [airflow] ad-m commented on issue #9808: Add tests for OpenAPI specification



ad-m commented on issue #9808:
URL: https://github.com/apache/airflow/issues/9808#issuecomment-658187916


   I was not aware that connection are able to validate responses:
   
https://github.com/apache/airflow/blob/53c0f2d987bf3dbfb59da283287f4aa7b72524ab/airflow/www/extensions/init_views.py#L105
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ad-m closed issue #9808: Add tests for OpenAPI specification



ad-m closed issue #9808:
URL: https://github.com/apache/airflow/issues/9808


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on a change in pull request #9819: Import Iterable from collections.abc in firestore operators



mik-laj commented on a change in pull request #9819:
URL: https://github.com/apache/airflow/pull/9819#discussion_r454366178



##
File path: airflow/providers/google/firebase/operators/firestore.py
##
@@ -15,7 +15,7 @@
 # specific language governing permissions and limitations
 # under the License.
 
-from collections import Iterable
+from collections.abc import Iterable

Review comment:
   It seems to me that here it will be better to use typing.Iterable. We 
only use it in types, not to create our own class.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on a change in pull request #9631: Add function to get current context



mik-laj commented on a change in pull request #9631:
URL: https://github.com/apache/airflow/pull/9631#discussion_r454363463



##
File path: airflow/models/taskinstance.py
##
@@ -1132,7 +1133,8 @@ def signal_handler(signum, frame):  # pylint: 
disable=unused-argument
 self._run_execute_callback(context, task)
 
 # Execute the task
-result = self._execute_task(context, task_copy)
+with set_current_context(context):

Review comment:
   Since it is used only functional DAG, why are we making changes to task 
instanace?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj commented on issue #9808: Add tests for OpenAPI specification



mik-laj commented on issue #9808:
URL: https://github.com/apache/airflow/issues/9808#issuecomment-658182931


   We use 
[connexion](https://connexion.readthedocs.io/en/latest/request.html#request-validation)
 to handle this case.  Do you want to add something more?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] mik-laj edited a comment on issue #9808: Add tests for OpenAPI specification



mik-laj edited a comment on issue #9808:
URL: https://github.com/apache/airflow/issues/9808#issuecomment-658182931


   We use 
[connexion](https://connexion.readthedocs.io/en/latest/request.html#request-validation)
 to handle validation for request body, response body, and parameters.  Do you 
want to add something more?



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on a change in pull request #9631: Add function to get current context



kaxil commented on a change in pull request #9631:
URL: https://github.com/apache/airflow/pull/9631#discussion_r454355273



##
File path: airflow/task/context/current.py
##
@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one

Review comment:
   yeah that might work, since it only applies to functional dags





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #9816: Helm chart can now place arbitrary config settings in to airflow.cfg



potiuk commented on pull request #9816:
URL: https://github.com/apache/airflow/pull/9816#issuecomment-658175454


   > @aneesh-joseph You can already customize the running config via env vars:
   > 
   > 
https://github.com/apache/airflow/blob/5ef332abdeba93c3877dde46a2693362b6feb273/chart/values.yaml#L94-L103
   > 
   > > Why don't we remove the ConfigMap completely and rely exclusively on 
environment variables?
   > 
   > @potiuk That wouldn't really change the "meat" of this PR, where we want 
some default configs set, but some of those defaults need to be dynamically 
computed.
   
   Agree. The meat of it is structuring the values.yml which I am super-happy 
about. What I was referring to as "unreadable" was the configmap.yaml :)
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on pull request #9816: Helm chart can now place arbitrary config settings in to airflow.cfg



ashb commented on pull request #9816:
URL: https://github.com/apache/airflow/pull/9816#issuecomment-658173163


   It's probably worth doing that and just clearly documenting that "These 
values are templated before use. Ensure anything is escaped".
   
   I'll update the PR.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] Datkros commented on pull request #9695: Add support for remote logging to be injected into the airflow.cfg configmap in helm chart



Datkros commented on pull request #9695:
URL: https://github.com/apache/airflow/pull/9695#issuecomment-658164002


   > > we'd lose visibility on what's the current configuration running by just 
looking at the source.
   > 
   > In Airflow 1.101.11, I added new command - `airflow config`. Will this 
help solve this problem?
   > 
   
   Not really, these commands are meant for adhoc configurations, not direct 
injections or automations, right? I
   
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] turbaszek opened a new pull request #9819: Import Iterable from collections.abc in firestore operators



turbaszek opened a new pull request #9819:
URL: https://github.com/apache/airflow/pull/9819


   DeprecationWarning: Using or importing the ABCs from 'collections'
   instead of from 'collections.abc' is deprecated since Python 3.3,
   and in 3.9 it will stop working
   
   ---
   Make sure to mark the boxes below before creating PR: [x]
   
   - [x] Description above provides context of the change
   - [x] Unit tests coverage for changes (not needed for documentation changes)
   - [x] Target Github ISSUE in description if exists
   - [x] Commits follow "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)"
   - [x] Relevant documentation is updated including usage instructions.
   - [x] I will engage committers as explained in [Contribution Workflow 
Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in 
[UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request 
Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)
 for more information.
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on pull request #9816: Helm chart can now place arbitrary config settings in to airflow.cfg



ashb commented on pull request #9816:
URL: https://github.com/apache/airflow/pull/9816#issuecomment-658162817


   > > I thought about using `tpl`, but then you have to worry about escaping 
in the Values.yml.
   > 
   > agree, but wouldn't something like this do the job?
   
   Yes, but my concern is that `tpl` could lead to confusing behaviour for 
users of the chart when trying set config values and getting odd/hard-to-debug 
errors.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] aneesh-joseph commented on pull request #9816: Helm chart can now place arbitrary config settings in to airflow.cfg



aneesh-joseph commented on pull request #9816:
URL: https://github.com/apache/airflow/pull/9816#issuecomment-658161779


   > I thought about using `tpl`, but then you have to worry about escaping in 
the Values.yml.
   
   agree, but wouldn't something like this do the job?
   
   ```
   airflow_cfg:
 core:
   dags_folder:  '{{ include "airflow_dags" . }}'
   load_examples:  false
   colored_console_log: false
   executor:  '{{ .Values.executor }}'
   remote_logging:  '{{- if .Values.elasticsearch.enabled }}true{{ else 
}}false{{ end }}'
 api:
   auth_backend:  '{{ .Values.api.authBackend }}'
 logging:
   logging_level: DEBUG
 webserver:
   enable_proxy_fix: true
   expose_config: true
   rbac: true
 celery:
   default_queue: celery
 scheduler:
   scheduler_heartbeat_sec: 5
   statsd_on:  '{{- if .Values.statsd.enabled }}true{{ else }}false{{ end 
}}'
   statsd_port: 9125
   statsd_prefix: airflow
   statsd_host:  '{{ printf "%s-statsd" .Release.Name }}'
   run_duration: 41460
 elasticsearch:
   json_format: true
   log_id_template: "{dag_id}_{task_id}_{execution_date}_{try_number}"
 elasticsearch_configs:
   max_retries: 3
   timeout: 30
   retry_timeout: True
 kubernetes:
   namespace:  '{{ .Release.Namespace }}'
   airflow_configmap:  '{{ include "airflow_config" . }}'
   airflow_local_settings_configmap:  '{{ include "airflow_config" . }}'
   worker_container_repository:  '{{ .Values.images.airflow.repository | 
default .Values.defaultAirflowRepository }}'
   worker_container_tag:  '{{ .Values.images.airflow.tag | default 
.Values.defaultAirflowTag }}'
   worker_container_image_pull_policy:  '{{ 
.Values.images.airflow.pullPolicy }}'
   worker_service_account_name:  '{{ .Release.Name }}-worker-serviceaccount'
   image_pull_secrets:  '{{ template "registry_secret" . }}'
   dags_in_image:  '{{- if or .Values.dags.gitSync.enabled 
.Values.dags.persistence.enabled }}False{{ else }}True{{ end }}'
   delete_worker_pods:  true
   run_as_user:  '{{ .Values.uid }}'
   fs_group:  '{{ .Values.gid }}'
   git_dags_folder_mount_point:  '{{- if or .Values.dags.gitSync.enabled 
.Values.dags.persistence.enabled }}{{ include "airflow_dags_mount_path" . 
}}{{end}}'
   dags_volume_mount_point:  '{{- if or .Values.dags.gitSync.enabled 
.Values.dags.persistence.enabled }}{{ include "airflow_dags_mount_path" . }}{{ 
end }}'
   dags_volume_claim:  '{{- if .Values.dags.persistence.enabled }}{{ 
include "airflow_dags_volume_claim" . }}{{ end }}'
   dags_volume_subpath:  '{{- if .Values.dags.persistence.enabled 
}}{{.Values.dags.gitSync.dest }}/{{ .Values.dags.gitSync.subPath }}{{ end }}'
   git_repo:  '{{- if and .Values.dags.gitSync.enabled (not 
.Values.dags.persistence.enabled) }}{{ .Values.dags.gitSync.repo }}{{ end }}'
   git_branch:  '{{ .Values.dags.gitSync.branch }}'
   git_sync_rev:  '{{ .Values.dags.gitSync.rev }}'
   git_sync_depth:  '{{ .Values.dags.gitSync.depth }}'
   git_sync_root:  '{{ .Values.dags.gitSync.root }}'
   git_sync_dest:  '{{ .Values.dags.gitSync.dest }}'
   git_sync_container_repository:  '{{ 
.Values.dags.gitSync.containerRepository }}'
   git_sync_container_tag:  '{{ .Values.dags.gitSync.containerTag }}'
   git_sync_init_container_name:  '{{ .Values.dags.gitSync.containerName }}'
   git_sync_run_as_user:  '{{ .Values.uid }}'
   git_ssh_known_hosts_configmap_name:  '{{- if  
.Values.dags.gitSync.knownHosts }}{{ include "airflow_config" . }}{{ end }}'
   git_ssh_key_secret_name:  '{{- if .Values.dags.gitSync.sshKeySecret }}{{ 
.Values.dags.gitSync.sshKeySecret }}{{ end }}'
   git_sync_credentials_secret:  '{{- if  
.Values.dags.gitSync.credentialsSecret }}{{ 
.Values.dags.gitSync.credentialsSecret }}{{ end }}'
 kubernetes_secrets:
   AIRFLOW__CORE__SQL_ALCHEMY_CONN:  '{{ printf "%s=connection" (include 
"airflow_metadata_secret" .) }}'
   AIRFLOW__CORE__FERNET_KEY:  '{{ printf "%s=fernet-key" (include 
"fernet_key_secret" .) }}'
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on a change in pull request #9817: S3 upload session complete sensor



ashb commented on a change in pull request #9817:
URL: https://github.com/apache/airflow/pull/9817#discussion_r454333273



##
File path: airflow/providers/amazon/aws/sensors/s3_upload_session_complete.py
##
@@ -0,0 +1,179 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+from datetime import datetime
+from typing import Optional, Set
+
+from airflow.exceptions import AirflowException
+from airflow.providers.amazon.aws.hooks.s3 import S3Hook
+from airflow.sensors.base_sensor_operator import BaseSensorOperator, 
poke_mode_only
+from airflow.utils.decorators import apply_defaults
+
+
+@poke_mode_only
+class S3UploadSessionCompleteSensor(BaseSensorOperator):
+"""
+Checks for changes in the number of objects at prefix in AWS S3
+bucket and returns True if the inactivity period has passed with no
+increase in the number of objects. Note, this sensor will no behave 
correctly
+in reschedule mode, as the state of the listed objects in the S3 bucket 
will
+be lost between rescheduled invocations.
+
+:param bucket_name: Name of the S3 bucket
+:type bucket_name: str
+:param prefix: The prefix being waited on. Relative path from bucket root 
level.
+:type prefix: str
+:param aws_conn_id: a reference to the s3 connection
+:type aws_conn_id: str
+:param verify: Whether or not to verify SSL certificates for S3 connection.
+By default SSL certificates are verified.
+You can provide the following values:
+
+- ``False``: do not validate SSL certificates. SSL will still be used
+ (unless use_ssl is False), but SSL certificates will not be
+ verified.
+- ``path/to/cert/bundle.pem``: A filename of the CA cert bundle to 
uses.
+ You can specify this argument if you want to use a different
+ CA cert bundle than the one used by botocore.
+:type verify: bool or str
+:param inactivity_period: The total seconds of inactivity to designate
+an upload session is over. Note, this mechanism is not real time and
+this operator may not return until a poke_interval after this period
+has passed with no additional objects sensed.
+:type inactivity_period: float
+:param min_objects: The minimum number of objects needed for upload session
+to be considered valid.
+:type min_objects: int
+:param previous_objects: The set of object ids found during the last poke.
+:type previous_objects: set[str]
+:param allow_delete: Should this sensor consider objects being deleted
+between pokes valid behavior. If true a warning message will be logged
+when this happens. If false an error will be raised.
+:type allow_delete: bool
+"""
+
+template_fields = ('bucket_name', 'prefix')
+
+@apply_defaults
+def __init__(self,
+ bucket_name: str,
+ prefix: str,
+ inactivity_period: float = 60 * 60,
+ min_objects: int = 1,
+ previous_objects: Optional[Set[str]] = None,
+ allow_delete: bool = True,
+ aws_conn_id: str = 'google_cloud_default',
+ delegate_to: Optional[str] = None,
+ *args, **kwargs) -> None:
+
+super().__init__(*args, **kwargs)
+
+self.bucket = bucket_name
+self.prefix = prefix
+if inactivity_period < 0:
+raise ValueError("inactivity_period must be non-negative")
+self.inactivity_period = inactivity_period
+self.min_objects = min_objects
+self.previous_objects = previous_objects if previous_objects else set()
+self.inactivity_seconds = 0
+self.allow_delete = allow_delete
+self.aws_conn_id = aws_conn_id
+self.delegate_to = delegate_to
+self.last_activity_time = None
+self.hook = None
+
+def _get_aws_hook(self):
+if not self.hook:
+self.hook = S3Hook()
+return self.hook
+
+def is_bucket_updated(self, current_objects: Set[str]) -> bool:
+"""
+Checks whether new objects have been uploaded and the inactivity_period
+has passed

[GitHub] [airflow] anikarni commented on a change in pull request #9817: S3 upload session complete sensor



anikarni commented on a change in pull request #9817:
URL: https://github.com/apache/airflow/pull/9817#discussion_r454333010



##
File path: airflow/providers/amazon/aws/sensors/s3_upload_session_complete.py
##
@@ -0,0 +1,179 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+from datetime import datetime
+from typing import Optional, Set
+
+from airflow.exceptions import AirflowException
+from airflow.providers.amazon.aws.hooks.s3 import S3Hook
+from airflow.sensors.base_sensor_operator import BaseSensorOperator, 
poke_mode_only
+from airflow.utils.decorators import apply_defaults
+
+
+@poke_mode_only
+class S3UploadSessionCompleteSensor(BaseSensorOperator):
+"""
+Checks for changes in the number of objects at prefix in AWS S3
+bucket and returns True if the inactivity period has passed with no
+increase in the number of objects. Note, this sensor will no behave 
correctly
+in reschedule mode, as the state of the listed objects in the S3 bucket 
will

Review comment:
   It does! The `@poke_mode_only` decorator should do the job! 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on a change in pull request #9817: S3 upload session complete sensor



ashb commented on a change in pull request #9817:
URL: https://github.com/apache/airflow/pull/9817#discussion_r454331148



##
File path: airflow/providers/amazon/aws/sensors/s3_upload_session_complete.py
##
@@ -0,0 +1,179 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+from datetime import datetime
+from typing import Optional, Set
+
+from airflow.exceptions import AirflowException
+from airflow.providers.amazon.aws.hooks.s3 import S3Hook
+from airflow.sensors.base_sensor_operator import BaseSensorOperator, 
poke_mode_only
+from airflow.utils.decorators import apply_defaults
+
+
+@poke_mode_only
+class S3UploadSessionCompleteSensor(BaseSensorOperator):
+"""
+Checks for changes in the number of objects at prefix in AWS S3
+bucket and returns True if the inactivity period has passed with no
+increase in the number of objects. Note, this sensor will no behave 
correctly
+in reschedule mode, as the state of the listed objects in the S3 bucket 
will

Review comment:
   It should throw an error then.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (AIRFLOW-7014) Operator for Apache Kylin



[ 
https://issues.apache.org/jira/browse/AIRFLOW-7014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157336#comment-17157336
 ] 

ASF GitHub Bot commented on AIRFLOW-7014:
-

liuyonghengheng commented on pull request #9149:
URL: https://github.com/apache/airflow/pull/9149#issuecomment-658156250


   @mik-laj @potiuk Hi Mik ,Hi Potiuk  Will you find some time to merge this pr 
?  This pr has ben blocked too long time, so that there are some conflicts . 
After I resolve these conflicts and rebase the master branch，the CI test can 
not pass。



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Operator for Apache Kylin 
> --
>
> Key: AIRFLOW-7014
> URL: https://issues.apache.org/jira/browse/AIRFLOW-7014
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: executors, operators
>Affects Versions: 1.10.0
>Reporter: Shao Feng Shi
>Assignee: liuyongheng
>Priority: Major
>
> Apache Kylin is an analytial data warehouse for big data. Kylin provides a 
> set of RESTful API for user to trigger the data loading, and also run SQL 
> queries against the OLAP cubes in sub-seconds latency. We developed a 
> KylinOperator within Airflow, so that user can easily trigger Kylin with 
> other tasks (hive, spark, etc), and plan to contribute into Airflow.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [airflow] liuyonghengheng commented on pull request #9149: [AIRFLOW-7014] Add Apache Kylin operator



liuyonghengheng commented on pull request #9149:
URL: https://github.com/apache/airflow/pull/9149#issuecomment-658156250


   @mik-laj @potiuk Hi Mik ,Hi Potiuk  Will you find some time to merge this pr 
?  This pr has ben blocked too long time, so that there are some conflicts . 
After I resolve these conflicts and rebase the master branch，the CI test can 
not pass。



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow-site] turbaszek opened a new pull request #273: Add new committers and PMCs



turbaszek opened a new pull request #273:
URL: https://github.com/apache/airflow-site/pull/273


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] kaxil commented on pull request #9816: Helm chart can now place arbitrary config settings in to airflow.cfg



kaxil commented on pull request #9816:
URL: https://github.com/apache/airflow/pull/9816#issuecomment-658143194


   Ya I feel the same way, code is very difficult to read.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on pull request #9816: Helm chart can now place arbitrary config settings in to airflow.cfg



ashb commented on pull request #9816:
URL: https://github.com/apache/airflow/pull/9816#issuecomment-658142386


   I agree though that this is _rather_ ugly. (It was going fine right up until 
I got to the monster that is the kubernetes section.)



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] ashb commented on pull request #9816: Helm chart can now place arbitrary config settings in to airflow.cfg



ashb commented on pull request #9816:
URL: https://github.com/apache/airflow/pull/9816#issuecomment-658142099


   @aneesh-joseph You can already customize the running config via env vars: 
https://github.com/apache/airflow/blob/5ef332abdeba93c3877dde46a2693362b6feb273/chart/values.yaml#L94-L103
   
   > Why don't we remove the ConfigMap completely and rely exclusively on 
environment variables?
   
   @potiuk That wouldn't really change the "meat" of this PR, where we want 
some default configs set, but some of those defaults need to be dynamically 
computed.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk commented on pull request #9816: Helm chart can now place arbitrary config settings in to airflow.cfg



potiuk commented on pull request #9816:
URL: https://github.com/apache/airflow/pull/9816#issuecomment-658140612


   > > We have also this airflow config command that we can always run at the 
end of initialization (in the init container ) to dump > the config in the 
"familiar" and nicely grouped way. We can even make it colored.
   > 
   > This would dump secrets/sensitive info to the logs, so shouldn't be on by 
default.
   
   Yeah. We could mask/remove secrets. But it's not needed at all to dump it at 
all - it's just a convenience method to show it in familiar way.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [airflow] potiuk edited a comment on pull request #9816: Helm chart can now place arbitrary config settings in to airflow.cfg