[GitHub] [airflow] potiuk commented on pull request #10280: Enable Sphinx spellcheck for docs generation
potiuk commented on pull request #10280: URL: https://github.com/apache/airflow/pull/10280#issuecomment-672112122 > @potiuk Do we need to bump the EPOCH number so that the CI uses the updated Dockerfile ? Absolutely not. It's only needed in case stuff is removed) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #10280: Enable Sphinx spellcheck for docs generation
kaxil commented on pull request #10280: URL: https://github.com/apache/airflow/pull/10280#issuecomment-672111000 @potiuk Do we need to bump the EPOCH number so that the CI uses the updated Dockerfile ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #10280: Enable Sphinx spellcheck for docs generation
potiuk commented on pull request #10280: URL: https://github.com/apache/airflow/pull/10280#issuecomment-672112284 And only for pip installs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] atsalolikhin-spokeo commented on issue #8388: Task instance details page blows up when no dagrun
atsalolikhin-spokeo commented on issue #8388: URL: https://github.com/apache/airflow/issues/8388#issuecomment-672153377 I ran into this in AWS, running Amazon Linux 2. `uname -a` shows: Linux ip-172-20-2-227.us-west-2.compute.internal 4.14.165-131.185.amzn2.x86_64 #1 SMP Wed Jan 15 14:19:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #10280: Enable Sphinx spellcheck for docs generation
kaxil commented on a change in pull request #10280: URL: https://github.com/apache/airflow/pull/10280#discussion_r468780812 ## File path: docs/spelling_wordlist.txt ## @@ -0,0 +1,1366 @@ +Ack +Acyclic +Airbnb +AirflowException +Aizhamal +Alphasort +Analytics +Anand +Ansible +AppBuilder +Arg +Args +Async +Auth +Autoscale +Avro +Azkaban +Azri +Backend +Backends +Backfill +BackfillJobTest +Backfills +Banco +Bas +BaseClient +BaseOperator +BaseView +Beauchemin +Behaviour +Bigquery +Bigtable +Bitshift +Bluecore +Bolke +Bool +Booleans +Boto +BounceX +Boxel +Bq +Breguła +CLoud +CSRFProtect +Cancelled +Cassanda +Catchup +Celect +Cgroups +Changelog +Chao +CheckOperator +Checklicence +Checkr +Cinimex +Cloudant +Cloudwatch +ClusterManagerClient +Codecov +Colour +ComputeNodeState +Computenodes +Config +Configs +Cron +Ctrl +Daemonize +DagFileProcessorManager +DagRun +Dagbag +Dagre +Dask +Databricks +Datadog +Dataflow +Dataframe +Datalake +Datanodes +Dataprep +Dataproc +Dataset +Datasets +Datastore +Datasync +DateFrame +Datetimepicker +Datetimes +Davydov +De +Decrypt +Decrypts +Deserialize +Deserialized +Dingding +Dlp +Dockerfile +Dockerhub +Dockerise +Docstring +Docstrings +Dont +Driesprong +Drivy +Dsn +Dynamodb +EDITMSG +ETag +Eg +EmrAddSteps +EmrCreateJobFlow +Enum +Env +Exasol +Failover +Feng +Fernet +FileSensor +Filebeat +Filepath +Fileshare +Fileshares +Filesystem +Firehose +Firestore +Fokko +Formaturas +Fundera +GCP +GCS +GH +GSoD +Gannt +Gantt +Gao +Gcp +Gentner +GetPartitions +GiB +Github +Gitter +Glassdoor +Gmail +Groupalia +Groupon +Grpc +Gunicorn +Guziel +Gzip +HCatalog +HTTPBasicAuth +Harenslak +Hashable +Hashicorp +Highcharts +Hitesh +Hiveserver +Hoc +Homan +Hostname +Hou +Http +HttpError +HttpRequest +IdP +Imap +Imberman +InsecureClient +InspectContentResponse +Investorise +JPype +Jakob +Jarek +Jdbc +Jiajie +Jinja +Jinjafied +Jinjafy +Jira +JobComplete +JobExists +JobRunning +Json +Jupyter +KYLIN +Kalibrr +Kamil +Kaxil +Kengo +Kerberized +Kerberos +KerberosClient +KevinYang +KeyManagementServiceClient +Keyfile +Kibana +Kinesis +Kombu +Kube +Kubernetes +Kusto +Kwargs +Kylin +LaunchTemplateParameters +Letsbonus +Lifecycle +LineItem +ListGenerator +Logstash +Lowin +Lyft +Maheshwari +Makefile +Mapreduce +Masternode +Maxime +Memorystore +Mesos +MessageAttributes +Metastore +Metatsore +Mixin +Mongo +Moto +Mysql +NFS +NaN +Naik +Namenode +Namespace +Nextdoor +Nones +NotFound +Nullable +Nurmamat +OAuth +Oauth +Oauthlib +Okta +Oozie +Opsgenie +Optimise +PEM +PTarget +Pagerduty +Papermill +Parallelize +Parameterizing +Paramiko +Params +Pem +Pinot +Popen +Postgres +Postgresql +Potiuk +Pre +Precommit +Preprocessed +Proc +Protobuf +PublisherClient +Pubsub +Py +PyPI +Pylint +Pyspark +PythonOperator +Qingping +Qplum +Quantopian +Qubole +Quboles +Quoble +RBAC +Readme +Realtime +Rebasing +Rebrand +Reddit +Redhat +Reinitialising +Retrives +Riccomini +Roadmap +Robinhood +SIGTERM +SSHClient +SSHTunnelForwarder +SaaS +Sagemaker +Sasl +SecretManagerClient +Seedlist +Seki +Sendgrid +Siddharth +SlackHook +SparkPi +SparkR +SparkSQL +Sql +Sqlalchemy +Sqlite +Sqoop +Stackdriver +Standarization +StatsD +Statsd +Stringified +Subclasses +Subdirectory +Submodules +Subpackages +Subpath +SubscriberClient +Subtasks +Sumit +Swtch +Systemd +TCP +TLS +TTY +TThe +TZ +TaskInstance +Taskfail +Templated +Templating +Teradata +TextToSpeechClient +Tez +Thinknear +ToC +Tomasz +Tooltip +Tsai +UA +UNload +Uellendall +Umask +Un +Undead +Undeads +Unpausing +Unpickle +Upsert +Upsight +Uptime +Urbaszek +Url +Utils +Vendorize +Vertica +Vevo +WTF +WaiterModel +Wasb +WebClient +Webhook +Webserver +Werkzeug +Wiedmer +XCom +XComs +Xcom +Xero +Xiaodong +Yamllint +Yandex +Yieldr +Zego +Zendesk +Zhong +Zsh +Zymergen +aIRFLOW +abc +accessor +accountmaking +acessible +ack +ackIds +acknowledgement +actionCard +acurate +acyclic +adhoc +aijamalnk +airbnb +airfl +airflowignore +ajax +alertPolicies +alexvanboxel +allAuthenticatedUsers +allUsers +allowinsert +analyse +analytics +analyticsreporting +analyzeEntities +analyzeSentiment +analyzeSyntax +aoen +apache +api +apikey +apis +appbuilder +approle +arg +args +arn +arraysize +artwr +asc +ascii +asciiart +asia +assertEqualIgnoreMultipleSpaces +ast +async +athena +attemping +attr +attrs +auth +authMechanism +authenication +authenticator +authorised +autoclass +autocommit +autocomplete +autodetect +autodetected +autoenv +autogenerated +autorestart +autoscale +autoscaling +avro +aws +awsbatch +backend +backends +backfill +backfilled +backfilling +backfills +backoff +backport +backreference +backtick +backticks +balancer +baseOperator +basedn +basestring +basetaskrunner +bashrc +basph +batchGet +bc +bcc +beeen +behaviour +behaviours +behavoiur +bicket +bigquery +bigtable +bitshift +bolkedebruin +booktabs +boolean +booleans +bootDiskType +boto +botocore +bowser +bq +bugfix +bugfixes +buildType +bytestring +cacert +callables +cancelled +carbonite +cas +cassandra +casted +catchup +cattrs +ccache +celeryd
[GitHub] [airflow] kaxil commented on a change in pull request #10280: Enable Sphinx spellcheck for docs generation
kaxil commented on a change in pull request #10280: URL: https://github.com/apache/airflow/pull/10280#discussion_r468780276 ## File path: docs/spelling_wordlist.txt ## @@ -0,0 +1,1366 @@ +Ack +Acyclic +Airbnb +AirflowException +Aizhamal +Alphasort +Analytics +Anand +Ansible +AppBuilder +Arg +Args +Async +Auth +Autoscale +Avro +Azkaban +Azri +Backend +Backends +Backfill +BackfillJobTest +Backfills +Banco +Bas +BaseClient +BaseOperator +BaseView +Beauchemin +Behaviour +Bigquery +Bigtable +Bitshift +Bluecore +Bolke +Bool +Booleans +Boto +BounceX +Boxel +Bq +Breguła +CLoud +CSRFProtect +Cancelled +Cassanda +Catchup +Celect +Cgroups +Changelog +Chao +CheckOperator +Checklicence +Checkr +Cinimex +Cloudant +Cloudwatch +ClusterManagerClient +Codecov +Colour +ComputeNodeState +Computenodes +Config +Configs +Cron +Ctrl +Daemonize +DagFileProcessorManager +DagRun +Dagbag +Dagre +Dask +Databricks +Datadog +Dataflow +Dataframe +Datalake +Datanodes +Dataprep +Dataproc +Dataset +Datasets +Datastore +Datasync +DateFrame +Datetimepicker +Datetimes +Davydov +De +Decrypt +Decrypts +Deserialize +Deserialized +Dingding +Dlp +Dockerfile +Dockerhub +Dockerise +Docstring +Docstrings +Dont +Driesprong +Drivy +Dsn +Dynamodb +EDITMSG +ETag +Eg +EmrAddSteps +EmrCreateJobFlow +Enum +Env +Exasol +Failover +Feng +Fernet +FileSensor +Filebeat +Filepath +Fileshare +Fileshares +Filesystem +Firehose +Firestore +Fokko +Formaturas +Fundera +GCP +GCS +GH +GSoD +Gannt +Gantt +Gao +Gcp +Gentner +GetPartitions +GiB +Github +Gitter +Glassdoor +Gmail +Groupalia +Groupon +Grpc +Gunicorn +Guziel +Gzip +HCatalog +HTTPBasicAuth +Harenslak +Hashable +Hashicorp +Highcharts +Hitesh +Hiveserver +Hoc +Homan +Hostname +Hou +Http +HttpError +HttpRequest +IdP +Imap +Imberman +InsecureClient +InspectContentResponse +Investorise +JPype +Jakob +Jarek +Jdbc +Jiajie +Jinja +Jinjafied +Jinjafy +Jira +JobComplete +JobExists +JobRunning +Json +Jupyter +KYLIN +Kalibrr +Kamil +Kaxil +Kengo +Kerberized +Kerberos +KerberosClient +KevinYang +KeyManagementServiceClient +Keyfile +Kibana +Kinesis +Kombu +Kube +Kubernetes +Kusto +Kwargs +Kylin +LaunchTemplateParameters +Letsbonus +Lifecycle +LineItem +ListGenerator +Logstash +Lowin +Lyft +Maheshwari +Makefile +Mapreduce +Masternode +Maxime +Memorystore +Mesos +MessageAttributes +Metastore +Metatsore +Mixin +Mongo +Moto +Mysql +NFS +NaN +Naik +Namenode +Namespace +Nextdoor +Nones +NotFound +Nullable +Nurmamat +OAuth +Oauth +Oauthlib +Okta +Oozie +Opsgenie +Optimise +PEM +PTarget +Pagerduty +Papermill +Parallelize +Parameterizing +Paramiko +Params +Pem +Pinot +Popen +Postgres +Postgresql +Potiuk +Pre +Precommit +Preprocessed +Proc +Protobuf +PublisherClient +Pubsub +Py +PyPI +Pylint +Pyspark +PythonOperator +Qingping +Qplum +Quantopian +Qubole +Quboles +Quoble +RBAC +Readme +Realtime +Rebasing +Rebrand +Reddit +Redhat +Reinitialising +Retrives +Riccomini +Roadmap +Robinhood +SIGTERM +SSHClient +SSHTunnelForwarder +SaaS +Sagemaker +Sasl +SecretManagerClient +Seedlist +Seki +Sendgrid +Siddharth +SlackHook +SparkPi +SparkR +SparkSQL +Sql +Sqlalchemy +Sqlite +Sqoop +Stackdriver +Standarization +StatsD +Statsd +Stringified +Subclasses +Subdirectory +Submodules +Subpackages +Subpath +SubscriberClient +Subtasks +Sumit +Swtch +Systemd +TCP +TLS +TTY +TThe +TZ +TaskInstance +Taskfail +Templated +Templating +Teradata +TextToSpeechClient +Tez +Thinknear +ToC +Tomasz +Tooltip +Tsai +UA +UNload +Uellendall +Umask +Un +Undead +Undeads +Unpausing +Unpickle +Upsert +Upsight +Uptime +Urbaszek +Url +Utils +Vendorize +Vertica +Vevo +WTF +WaiterModel +Wasb +WebClient +Webhook +Webserver +Werkzeug +Wiedmer +XCom +XComs +Xcom +Xero +Xiaodong +Yamllint +Yandex +Yieldr +Zego +Zendesk +Zhong +Zsh +Zymergen +aIRFLOW +abc +accessor +accountmaking +acessible +ack +ackIds +acknowledgement +actionCard +acurate +acyclic +adhoc +aijamalnk +airbnb +airfl +airflowignore +ajax +alertPolicies +alexvanboxel +allAuthenticatedUsers +allUsers +allowinsert +analyse +analytics +analyticsreporting +analyzeEntities +analyzeSentiment +analyzeSyntax +aoen +apache +api +apikey +apis +appbuilder +approle +arg +args +arn +arraysize +artwr +asc +ascii +asciiart +asia +assertEqualIgnoreMultipleSpaces +ast +async +athena +attemping +attr +attrs +auth +authMechanism +authenication +authenticator +authorised +autoclass +autocommit +autocomplete +autodetect +autodetected +autoenv +autogenerated +autorestart +autoscale +autoscaling +avro +aws +awsbatch +backend +backends +backfill +backfilled +backfilling +backfills +backoff +backport +backreference +backtick +backticks +balancer +baseOperator +basedn +basestring +basetaskrunner +bashrc +basph +batchGet +bc +bcc +beeen +behaviour +behaviours +behavoiur +bicket +bigquery +bigtable +bitshift +bolkedebruin +booktabs +boolean +booleans +bootDiskType +boto +botocore +bowser +bq +bugfix +bugfixes +buildType +bytestring +cacert +callables +cancelled +carbonite +cas +cassandra +casted +catchup +cattrs +ccache +celeryd
[GitHub] [airflow] atsalolikhin-spokeo commented on issue #8388: Task instance details page blows up when no dagrun
atsalolikhin-spokeo commented on issue #8388:
URL: https://github.com/apache/airflow/issues/8388#issuecomment-672169136
Thank you for re-opening this issue. I've checked the master version:
```
[airflow@ip-172-20-2-227 airflow]$ airflow version
2.0.0.dev0
[airflow@ip-172-20-2-227 airflow]$
```
I can't login through the Web UI, though. It's asking me for username and
password.
When I started the webserver, it said:
[2020-08-11 18:25:27,417] {manager.py:719} WARNING - No user yet created,
use flask fab command to do it.
I tried, but it didn't work. Help?
```
[airflow@ip-172-20-2-227 airflow]$ flask fab create-user
Role [Public]:
Username: admin
User first name: admin
User last name: admin
Email: ad...@example.com
Password:
Repeat for confirmation:
Usage: flask fab create-user [OPTIONS]
Error: Could not locate a Flask application. You did not provide the
"FLASK_APP" environment variable, and a "wsgi.py" or "app.py" module was not
found in the current directory.
[airflow@ip-172-20-2-227 airflow]$
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] atsalolikhin-spokeo commented on issue #8388: Task instance details page blows up when no dagrun
atsalolikhin-spokeo commented on issue #8388: URL: https://github.com/apache/airflow/issues/8388#issuecomment-672185365 I see this on my console: ``` Please make sure to build the frontend in static/ directory and restart the server ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] vlasvlasvlas commented on pull request #9715: create CODE_OF_CONDUCT.md
vlasvlasvlas commented on pull request #9715: URL: https://github.com/apache/airflow/pull/9715#issuecomment-672185359 yeayyy! :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ryanrichholt commented on pull request #6317: [AIRFLOW-5644] Simplify TriggerDagRunOperator usage
ryanrichholt commented on pull request #6317: URL: https://github.com/apache/airflow/pull/6317#issuecomment-672277149 In the DagRun model list view there is an option to search by run_id. Since we set the run_id when the dag is triggered externally, this field can be used when we need to check on progress for a specific case. This seems to work alright, but there are some downsides: - run_id doesn't propagate to subdags, so we basically cant use subdags - run_id can't be set when triggering the dag from some of the quick buttons in the ui - with the create dag_run form, the run_id can be set, but the conf cannot So, I'm also exploring options for moving away from using run_ids, and just adding this info to the run conf. But, I think this is going to require updating the DagRun model views, or a custom view plugin. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (AIRFLOW-5644) Simplify TriggerDagRunOperator usage
[ https://issues.apache.org/jira/browse/AIRFLOW-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175840#comment-17175840 ] ASF GitHub Bot commented on AIRFLOW-5644: - ryanrichholt commented on pull request #6317: URL: https://github.com/apache/airflow/pull/6317#issuecomment-672277149 In the DagRun model list view there is an option to search by run_id. Since we set the run_id when the dag is triggered externally, this field can be used when we need to check on progress for a specific case. This seems to work alright, but there are some downsides: - run_id doesn't propagate to subdags, so we basically cant use subdags - run_id can't be set when triggering the dag from some of the quick buttons in the ui - with the create dag_run form, the run_id can be set, but the conf cannot So, I'm also exploring options for moving away from using run_ids, and just adding this info to the run conf. But, I think this is going to require updating the DagRun model views, or a custom view plugin. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Simplify TriggerDagRunOperator usage > > > Key: AIRFLOW-5644 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5644 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Priority: Major > Fix For: 2.0.0 > > > The TriggerDagRunOperator usage is rather odd at the moment, especially the > way to pass a conf. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [airflow] maganaluis commented on pull request #8256: updated _write_args on PythonVirtualenvOperator
maganaluis commented on pull request #8256: URL: https://github.com/apache/airflow/pull/8256#issuecomment-672291543 @feluelle Sorry for the late response, should be up to date now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] darwinyip commented on issue #10275: Trigger rule for "always"
darwinyip commented on issue #10275: URL: https://github.com/apache/airflow/issues/10275#issuecomment-672091385 That's true. Closing issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #8388: Task instance details page blows up when no dagrun
mik-laj commented on issue #8388: URL: https://github.com/apache/airflow/issues/8388#issuecomment-672213162 Is the issue discussed in this ticket still present in this version? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (AIRFLOW-3964) Consolidate and de-duplicate sensor tasks
[
https://issues.apache.org/jira/browse/AIRFLOW-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175710#comment-17175710
]
ASF GitHub Bot commented on AIRFLOW-3964:
-
YingboWang commented on a change in pull request #5499:
URL: https://github.com/apache/airflow/pull/5499#discussion_r468732624
##
File path: airflow/models/taskinstance.py
##
@@ -280,7 +281,8 @@ def try_number(self):
database, in all other cases this will be incremented.
"""
# This is designed so that task logs end up in the right file.
-if self.state == State.RUNNING:
+# TODO: whether we need sensing here or not (in sensor and
task_instance state machine)
+if self.state in State.running():
Review comment:
"sensing" state means this TI is currently being poked by smart sensor.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Consolidate and de-duplicate sensor tasks
> --
>
> Key: AIRFLOW-3964
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3964
> Project: Apache Airflow
> Issue Type: Improvement
> Components: dependencies, operators, scheduler
>Affects Versions: 1.10.0
>Reporter: Yingbo Wang
>Assignee: Yingbo Wang
>Priority: Critical
>
> h2. Problem
> h3. Airflow Sensor:
> Sensors are a certain type of operator that will keep running until a certain
> criterion is met. Examples include a specific file landing in HDFS or S3, a
> partition appearing in Hive, or a specific time of the day. Sensors are
> derived from BaseSensorOperator and run a poke method at a specified
> poke_interval until it returns True.
> Airflow Sensor duplication is a normal problem for large scale airflow
> project. There are duplicated partitions needing to be detected from
> same/different DAG. In Airbnb there are 88 boxes running four different types
> of sensors everyday. The number of running sensor tasks ranges from 8k to
> 16k, which takes great amount of resources. Although Airflow team had
> redirected all sensors to a specific queue to allocate relatively minor
> resource, there is still large room to reduce the number of workers and
> relief DB pressure by optimizing the sensor mechanism.
> Existing sensor implementation creates an identical task for any sensor task
> with specific dag_id, task_id and execution_date. This task is responsible of
> keeping querying DB until the specified partitions exists. Even if two tasks
> are waiting for same partition in DB, they are creating two connections with
> the DB and checking the status in two separate processes. In one hand, DB
> need to run duplicate jobs in multiple processes which will take both cpu and
> memory resources. At the same time, Airflow need to maintain a process for
> each sensor to query and wait for the partition/table to be created.
> h1. ***Design*
> There are several issues need to be resolved for our smart sensor.
> h2. Persist sensor infor in DB and avoid file parsing before running
> Current Airflow implementation need to parse the DAG python file before
> running a task. Parsing multiple python file in a smart sensor make the case
> low efficiency and overload. Since sensor tasks need relatively more “light
> weight” executing information -- less number of properties with simple
> structure (most are built in type instead of function or object). We propose
> to skip the parsing for smart sensor. The easiest way is to persist all task
> instance information in airflow metaDB.
> h3. Solution:
> It will be hard to dump the whole task instance object dictionary. And we do
> not really need that much information.
> We add two sets to the base sensor class as “persist_fields” and
> “execute_fields”.
> h4. “persist_fields” dump to airflow.task_instance column “attr_dict”
> saves the attribute names that should be used to accomplish a sensor poking
> job. For example:
> # the “NamedHivePartitionSensor” define its persist_fields as
> ('partition_names', 'metastore_conn_id', 'hook') since these properties are
> enough for its poking function.
> # While the HivePartitionSensor can be slightly different use persist_fields
> as ('schema', 'table', 'partition', 'metastore_conn_id')
> If we have two tasks that have same property value for all field in
> persist_fields. That means these two tasks are poking the same item and they
> are holding duplicate poking jobs in senser.
> *The persist_fields can help us in deduplicate sensor tasks*. In a more
> broader way. If we can list
[GitHub] [airflow] YingboWang commented on a change in pull request #5499: [AIRFLOW-3964][AIP-17] Build smart sensor
YingboWang commented on a change in pull request #5499: URL: https://github.com/apache/airflow/pull/5499#discussion_r468732624 ## File path: airflow/models/taskinstance.py ## @@ -280,7 +281,8 @@ def try_number(self): database, in all other cases this will be incremented. """ # This is designed so that task logs end up in the right file. -if self.state == State.RUNNING: +# TODO: whether we need sensing here or not (in sensor and task_instance state machine) +if self.state in State.running(): Review comment: "sensing" state means this TI is currently being poked by smart sensor. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #10280: Enable Sphinx spellcheck for docs generation
kaxil commented on pull request #10280: URL: https://github.com/apache/airflow/pull/10280#issuecomment-672117470 > > @potiuk Do we need to bump the EPOCH number so that the CI uses the updated Dockerfile ? > > Absolutely not. It's only needed in case stuff is removed) Any ideas on how we fix the failing test: https://github.com/apache/airflow/pull/10280/checks?check_run_id=969045276#step:5:6023 ``` THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them. ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] Labbs opened a new issue #10292: Kubernetes Executor - invalid hostname in the database
Labbs opened a new issue #10292:
URL: https://github.com/apache/airflow/issues/10292
**Apache Airflow version**: 1.10.11
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
```
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1",
GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"clean",
BuildDate:"2020-04-10T21:53:51Z", GoVersion:"go1.14.2", Compiler:"gc",
Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+",
GitVersion:"v1.17.6-eks-4e7f64",
GitCommit:"4e7f642f9f4cbb3c39a4fc6ee84fe341a8ade94c", GitTreeState:"clean",
BuildDate:"2020-06-11T13:55:35Z", GoVersion:"go1.13.9", Compiler:"gc",
Platform:"linux/amd64"}
```
**Environment**:
- **Cloud provider or hardware configuration**: AWS
**What happened**:
When I want to retrieve the logs from the GUI, Airflow indicates that it has
not found the pod
```
HTTP response body:
b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods
\\"examplekubernetesexecutorstarttask-7409bf0f1ed644cfba6c423be769\\" not
found","reason":"NotFound","details":{"name":"examplekubernetesexecutorstarttask-7409bf0f1ed644cfba6c423be769","kind":"pods"},"code":404}\n'
```
After checking in the kubernetes and the database, the hostname is in the
database is not complete.
```
kubectl get po -n airflow-testing | grep
examplekubernetesexecutorstarttask-7409bf0f1ed644cfba6c423be769
examplekubernetesexecutorstarttask-7409bf0f1ed644cfba6c423be769a8c4
0/1 Completed 0 107m
```
It's missing 4 characters in the hostname.
If I update the database with the complete hostname, airflow retrieve the
log without issue
**What you expected to happen**:
The hostname must be complete in the database so that airflow can retrieve
the logs in kubernetes with the correct pod name.
**How to reproduce it**:
1- configure airflow for use KubernetesExecutor
```
AIRFLOW__KUBERNETES__DELETE_WORKER_PODS=False
AIRFLOW__CORE__EXECUTOR=KubernetesExecutor
```
2- add dag
```
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from airflow.utils.dates import days_ago
args = {
"owner": "airflow",
}
def print_stuff():
print("ok")
dag = DAG(
dag_id="example_kubernetes_executor",
default_args=args,
schedule_interval=None,
start_date=days_ago(2),
tags=["example"],
)
start_task = PythonOperator(
dag=dag,
task_id="start_task",
python_callable=print_stuff)
one_task = PythonOperator(
dag=dag,
task_id="one_task",
python_callable=print_stuff,
executor_config={"KubernetesExecutor": {"image":
"apache/airflow:latest"}},
)
start_task >> one_task
```
3- trigger execution
4- get logs from UI
Result:
```
*** Trying to get logs (last 100 lines) from worker pod
examplekubernetesexecutorstarttask-7409bf0f1ed644cfba6c423be769 ***
*** Unable to fetch logs from worker pod
examplekubernetesexecutorstarttask-7409bf0f1ed644cfba6c423be769 ***
(404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id':
'28495ca5-86f3-4935-9cef-f825b79cdff6', 'Cache-Control': 'no-cache, private',
'Content-Type': 'application/json', 'Date': 'Tue, 11 Aug 2020 16:27:43 GMT',
'Content-Length': '294'})
HTTP response body:
b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods
\\"examplekubernetesexecutorstarttask-7409bf0f1ed644cfba6c423be769\\" not
found","reason":"NotFound","details":{"name":"examplekubernetesexecutorstarttask-7409bf0f1ed644cfba6c423be769","kind":"pods"},"code":404}\n'
```
**Anything else we need to know**:
The problem is permanent regardless of the dag used.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #10280: Enable Sphinx spellcheck for docs generation
kaxil commented on a change in pull request #10280: URL: https://github.com/apache/airflow/pull/10280#discussion_r468780072 ## File path: docs/spelling_wordlist.txt ## @@ -0,0 +1,1366 @@ +Ack +Acyclic +Airbnb +AirflowException +Aizhamal +Alphasort +Analytics +Anand +Ansible +AppBuilder +Arg +Args +Async +Auth +Autoscale +Avro +Azkaban +Azri +Backend +Backends +Backfill +BackfillJobTest +Backfills +Banco +Bas +BaseClient +BaseOperator +BaseView +Beauchemin +Behaviour +Bigquery +Bigtable +Bitshift +Bluecore +Bolke +Bool +Booleans +Boto +BounceX +Boxel +Bq +Breguła +CLoud +CSRFProtect +Cancelled +Cassanda +Catchup +Celect +Cgroups +Changelog +Chao +CheckOperator +Checklicence +Checkr +Cinimex +Cloudant +Cloudwatch +ClusterManagerClient +Codecov +Colour +ComputeNodeState +Computenodes +Config +Configs +Cron +Ctrl +Daemonize +DagFileProcessorManager +DagRun +Dagbag +Dagre +Dask +Databricks +Datadog +Dataflow +Dataframe +Datalake +Datanodes +Dataprep +Dataproc +Dataset +Datasets +Datastore +Datasync +DateFrame +Datetimepicker +Datetimes +Davydov +De +Decrypt +Decrypts +Deserialize +Deserialized +Dingding +Dlp +Dockerfile +Dockerhub +Dockerise +Docstring +Docstrings +Dont +Driesprong +Drivy +Dsn +Dynamodb +EDITMSG +ETag +Eg +EmrAddSteps +EmrCreateJobFlow +Enum +Env +Exasol +Failover +Feng +Fernet +FileSensor +Filebeat +Filepath +Fileshare +Fileshares +Filesystem +Firehose +Firestore +Fokko +Formaturas +Fundera +GCP +GCS +GH +GSoD +Gannt +Gantt +Gao +Gcp +Gentner +GetPartitions +GiB +Github +Gitter +Glassdoor +Gmail +Groupalia +Groupon +Grpc +Gunicorn +Guziel +Gzip +HCatalog +HTTPBasicAuth +Harenslak +Hashable +Hashicorp +Highcharts +Hitesh +Hiveserver +Hoc +Homan +Hostname +Hou +Http +HttpError +HttpRequest +IdP +Imap +Imberman +InsecureClient +InspectContentResponse +Investorise +JPype +Jakob +Jarek +Jdbc +Jiajie +Jinja +Jinjafied +Jinjafy +Jira +JobComplete +JobExists +JobRunning +Json +Jupyter +KYLIN +Kalibrr +Kamil +Kaxil +Kengo +Kerberized +Kerberos +KerberosClient +KevinYang +KeyManagementServiceClient +Keyfile +Kibana +Kinesis +Kombu +Kube +Kubernetes +Kusto +Kwargs +Kylin +LaunchTemplateParameters +Letsbonus +Lifecycle +LineItem +ListGenerator +Logstash +Lowin +Lyft +Maheshwari +Makefile +Mapreduce +Masternode +Maxime +Memorystore +Mesos +MessageAttributes +Metastore +Metatsore +Mixin +Mongo +Moto +Mysql +NFS +NaN +Naik +Namenode +Namespace +Nextdoor +Nones +NotFound +Nullable +Nurmamat +OAuth +Oauth +Oauthlib +Okta +Oozie +Opsgenie +Optimise +PEM +PTarget +Pagerduty +Papermill +Parallelize +Parameterizing +Paramiko +Params +Pem +Pinot +Popen +Postgres +Postgresql +Potiuk +Pre +Precommit +Preprocessed +Proc +Protobuf +PublisherClient +Pubsub +Py +PyPI +Pylint +Pyspark +PythonOperator +Qingping +Qplum +Quantopian +Qubole +Quboles +Quoble +RBAC +Readme +Realtime +Rebasing +Rebrand +Reddit +Redhat +Reinitialising +Retrives +Riccomini +Roadmap +Robinhood +SIGTERM +SSHClient +SSHTunnelForwarder +SaaS +Sagemaker +Sasl +SecretManagerClient +Seedlist +Seki +Sendgrid +Siddharth +SlackHook +SparkPi +SparkR +SparkSQL +Sql +Sqlalchemy +Sqlite +Sqoop +Stackdriver +Standarization +StatsD +Statsd +Stringified +Subclasses +Subdirectory +Submodules +Subpackages +Subpath +SubscriberClient +Subtasks +Sumit +Swtch +Systemd +TCP +TLS +TTY +TThe +TZ +TaskInstance +Taskfail +Templated +Templating +Teradata +TextToSpeechClient +Tez +Thinknear +ToC +Tomasz +Tooltip +Tsai +UA +UNload +Uellendall +Umask +Un +Undead +Undeads +Unpausing +Unpickle +Upsert +Upsight +Uptime +Urbaszek +Url +Utils +Vendorize +Vertica +Vevo +WTF +WaiterModel +Wasb +WebClient +Webhook +Webserver +Werkzeug +Wiedmer +XCom +XComs +Xcom +Xero +Xiaodong +Yamllint +Yandex +Yieldr +Zego +Zendesk +Zhong +Zsh +Zymergen +aIRFLOW +abc +accessor +accountmaking +acessible +ack +ackIds +acknowledgement +actionCard +acurate +acyclic +adhoc +aijamalnk +airbnb +airfl +airflowignore +ajax +alertPolicies +alexvanboxel +allAuthenticatedUsers +allUsers +allowinsert +analyse +analytics +analyticsreporting +analyzeEntities +analyzeSentiment +analyzeSyntax +aoen +apache +api +apikey +apis +appbuilder +approle +arg +args +arn +arraysize +artwr +asc +ascii +asciiart +asia +assertEqualIgnoreMultipleSpaces +ast +async +athena +attemping +attr +attrs +auth +authMechanism +authenication +authenticator +authorised +autoclass +autocommit +autocomplete +autodetect +autodetected +autoenv +autogenerated +autorestart +autoscale +autoscaling +avro +aws +awsbatch +backend +backends +backfill +backfilled +backfilling +backfills +backoff +backport +backreference +backtick +backticks +balancer +baseOperator +basedn +basestring +basetaskrunner +bashrc +basph +batchGet +bc +bcc +beeen +behaviour +behaviours +behavoiur +bicket +bigquery +bigtable +bitshift +bolkedebruin +booktabs +boolean +booleans +bootDiskType +boto +botocore +bowser +bq +bugfix +bugfixes +buildType +bytestring +cacert +callables +cancelled +carbonite +cas +cassandra +casted +catchup +cattrs +ccache +celeryd
[GitHub] [airflow] kaxil commented on a change in pull request #10280: Enable Sphinx spellcheck for docs generation
kaxil commented on a change in pull request #10280: URL: https://github.com/apache/airflow/pull/10280#discussion_r468779562 ## File path: docs/spelling_wordlist.txt ## @@ -0,0 +1,1366 @@ +Ack +Acyclic +Airbnb +AirflowException +Aizhamal +Alphasort +Analytics +Anand +Ansible +AppBuilder +Arg +Args +Async +Auth +Autoscale +Avro +Azkaban +Azri +Backend +Backends +Backfill +BackfillJobTest +Backfills +Banco +Bas +BaseClient +BaseOperator +BaseView +Beauchemin +Behaviour +Bigquery +Bigtable +Bitshift +Bluecore +Bolke +Bool +Booleans +Boto +BounceX +Boxel +Bq +Breguła +CLoud +CSRFProtect +Cancelled +Cassanda +Catchup +Celect +Cgroups +Changelog +Chao +CheckOperator +Checklicence +Checkr +Cinimex +Cloudant +Cloudwatch +ClusterManagerClient +Codecov +Colour +ComputeNodeState +Computenodes +Config +Configs +Cron +Ctrl +Daemonize +DagFileProcessorManager +DagRun +Dagbag +Dagre +Dask +Databricks +Datadog +Dataflow +Dataframe +Datalake +Datanodes +Dataprep +Dataproc +Dataset +Datasets +Datastore +Datasync +DateFrame +Datetimepicker +Datetimes +Davydov +De +Decrypt +Decrypts +Deserialize +Deserialized +Dingding +Dlp +Dockerfile +Dockerhub +Dockerise +Docstring +Docstrings +Dont +Driesprong +Drivy +Dsn +Dynamodb +EDITMSG +ETag +Eg +EmrAddSteps +EmrCreateJobFlow +Enum +Env +Exasol +Failover +Feng +Fernet +FileSensor +Filebeat +Filepath +Fileshare +Fileshares +Filesystem +Firehose +Firestore +Fokko +Formaturas +Fundera +GCP +GCS +GH +GSoD +Gannt +Gantt +Gao +Gcp +Gentner +GetPartitions +GiB +Github +Gitter +Glassdoor +Gmail +Groupalia +Groupon +Grpc +Gunicorn +Guziel +Gzip +HCatalog +HTTPBasicAuth +Harenslak +Hashable +Hashicorp +Highcharts +Hitesh +Hiveserver +Hoc +Homan +Hostname +Hou +Http +HttpError +HttpRequest +IdP +Imap +Imberman +InsecureClient +InspectContentResponse +Investorise +JPype +Jakob +Jarek +Jdbc +Jiajie +Jinja +Jinjafied +Jinjafy +Jira +JobComplete +JobExists +JobRunning +Json +Jupyter +KYLIN +Kalibrr +Kamil +Kaxil +Kengo +Kerberized +Kerberos +KerberosClient +KevinYang +KeyManagementServiceClient +Keyfile +Kibana +Kinesis +Kombu +Kube +Kubernetes +Kusto +Kwargs +Kylin +LaunchTemplateParameters +Letsbonus +Lifecycle +LineItem +ListGenerator +Logstash +Lowin +Lyft +Maheshwari +Makefile +Mapreduce +Masternode +Maxime +Memorystore +Mesos +MessageAttributes +Metastore +Metatsore +Mixin +Mongo +Moto +Mysql +NFS +NaN +Naik +Namenode +Namespace +Nextdoor +Nones +NotFound +Nullable +Nurmamat +OAuth +Oauth +Oauthlib +Okta +Oozie +Opsgenie +Optimise +PEM +PTarget +Pagerduty +Papermill +Parallelize +Parameterizing +Paramiko +Params +Pem +Pinot +Popen +Postgres +Postgresql +Potiuk +Pre +Precommit +Preprocessed +Proc +Protobuf +PublisherClient +Pubsub +Py +PyPI +Pylint +Pyspark +PythonOperator +Qingping +Qplum +Quantopian +Qubole +Quboles +Quoble +RBAC +Readme +Realtime +Rebasing +Rebrand +Reddit +Redhat +Reinitialising +Retrives +Riccomini +Roadmap +Robinhood +SIGTERM +SSHClient +SSHTunnelForwarder +SaaS +Sagemaker +Sasl +SecretManagerClient +Seedlist +Seki +Sendgrid +Siddharth +SlackHook +SparkPi +SparkR +SparkSQL +Sql +Sqlalchemy +Sqlite +Sqoop +Stackdriver +Standarization +StatsD +Statsd +Stringified +Subclasses +Subdirectory +Submodules +Subpackages +Subpath +SubscriberClient +Subtasks +Sumit +Swtch +Systemd +TCP +TLS +TTY +TThe +TZ +TaskInstance +Taskfail +Templated +Templating +Teradata +TextToSpeechClient +Tez +Thinknear +ToC +Tomasz +Tooltip +Tsai +UA +UNload +Uellendall +Umask +Un +Undead +Undeads +Unpausing +Unpickle +Upsert +Upsight +Uptime +Urbaszek +Url +Utils +Vendorize +Vertica +Vevo +WTF +WaiterModel +Wasb +WebClient +Webhook +Webserver +Werkzeug +Wiedmer +XCom +XComs +Xcom +Xero +Xiaodong +Yamllint +Yandex +Yieldr +Zego +Zendesk +Zhong +Zsh +Zymergen +aIRFLOW +abc +accessor +accountmaking +acessible +ack +ackIds +acknowledgement +actionCard +acurate +acyclic +adhoc +aijamalnk +airbnb +airfl +airflowignore +ajax +alertPolicies +alexvanboxel +allAuthenticatedUsers +allUsers +allowinsert +analyse +analytics +analyticsreporting +analyzeEntities +analyzeSentiment +analyzeSyntax +aoen +apache +api +apikey +apis +appbuilder +approle +arg +args +arn +arraysize +artwr +asc +ascii +asciiart +asia +assertEqualIgnoreMultipleSpaces +ast +async +athena +attemping +attr +attrs +auth +authMechanism +authenication +authenticator +authorised +autoclass +autocommit +autocomplete +autodetect +autodetected +autoenv +autogenerated +autorestart +autoscale +autoscaling +avro +aws +awsbatch +backend +backends +backfill +backfilled +backfilling +backfills +backoff +backport +backreference +backtick +backticks +balancer +baseOperator +basedn +basestring +basetaskrunner +bashrc +basph +batchGet +bc +bcc +beeen +behaviour +behaviours +behavoiur +bicket +bigquery +bigtable +bitshift +bolkedebruin +booktabs +boolean +booleans +bootDiskType +boto +botocore +bowser Review comment: This actually is a typo. It is used in `privacy_notice.rst` -> It should be `browser`
[GitHub] [airflow] boring-cyborg[bot] commented on issue #10292: Kubernetes Executor - invalid hostname in the database
boring-cyborg[bot] commented on issue #10292: URL: https://github.com/apache/airflow/issues/10292#issuecomment-672167998 Thanks for opening your first issue here! Be sure to follow the issue template! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] houqp commented on a change in pull request #10267: Add Authentication for Stable API
houqp commented on a change in pull request #10267: URL: https://github.com/apache/airflow/pull/10267#discussion_r468873335 ## File path: airflow/api_connexion/endpoints/dag_run_endpoint.py ## @@ -151,6 +156,7 @@ def get_dag_runs_batch(session): total_entries=total_entries)) +@security.requires_authentication Review comment: if no auth is the exception here (i.e. only /health and /version), wouldn't it be safer to make auth required the default, then turn off auth specifically for those two endpoints? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #10280: Enable Sphinx spellcheck for docs generation
kaxil commented on pull request #10280: URL: https://github.com/apache/airflow/pull/10280#issuecomment-672130815 Got it, I restarted the tests again 爛 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #8388: Task instance details page blows up when no dagrun
mik-laj commented on issue #8388: URL: https://github.com/apache/airflow/issues/8388#issuecomment-672155750 This fix may not have been officially released. Can you check to see if the problem is in the master version? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] nadflinn opened a new issue #8388: Task instance details page blows up when no dagrun
nadflinn opened a new issue #8388: URL: https://github.com/apache/airflow/issues/8388 **Apache Airflow version**: 1.10.9 although I think it would apply to 1.10.10, 1.10.8, 1.10.7, and 1.10.6 **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): **Environment**: - **Cloud provider or hardware configuration**: aws - **OS** (e.g. from /etc/os-release): Ubuntu 18.04.3 LTS - **Kernel** (e.g. `uname -a`): `Linux 567ff872fc2d 3.13.0-170-generic #220-Ubuntu SMP Thu May 9 12:40:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux` - **Install tools**: - **Others**: **What happened**: If you have a DAG that has not been turned on before and you click on that DAG in the admin page and then in the DAG graph click on one of the tasks and then click on `Task Instance Details` you get the "Ooops" explosion error page. The error is: ``` Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 2447, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1952, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1821, in handle_user_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python3.7/dist-packages/flask/_compat.py", line 39, in reraise raise value File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1950, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1936, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/usr/local/lib/python3.7/dist-packages/flask_admin/base.py", line 69, in inner return self._run_view(f, *args, **kwargs) File "/usr/local/lib/python3.7/dist-packages/flask_admin/base.py", line 368, in _run_view return fn(self, *args, **kwargs) File "/usr/local/lib/python3.7/dist-packages/flask_login/utils.py", line 258, in decorated_view return func(*args, **kwargs) File "/usr/local/lib/python3.7/dist-packages/airflow/www/utils.py", line 290, in wrapper return f(*args, **kwargs) File "/usr/local/lib/python3.7/dist-packages/airflow/www/views.py", line 1041, in task dep_context=dep_context)] File "/usr/local/lib/python3.7/dist-packages/airflow/www/views.py", line 1039, in failed_dep_reasons = [(dep.dep_name, dep.reason) for dep in File "/usr/local/lib/python3.7/dist-packages/airflow/models/taskinstance.py", line 668, in get_failed_dep_statuses dep_context): File "/usr/local/lib/python3.7/dist-packages/airflow/ti_deps/deps/base_ti_dep.py", line 106, in get_dep_statuses for dep_status in self._get_dep_statuses(ti, session, dep_context): File "/usr/local/lib/python3.7/dist-packages/airflow/ti_deps/deps/dagrun_id_dep.py", line 51, in _get_dep_statuses if not dagrun.run_id or not match(BackfillJob.ID_PREFIX + '.*', dagrun.run_id): AttributeError: 'NoneType' object has no attribute 'run_id' ``` **What you expected to happen**: No error page. It should have loaded the standard task instance details page. I believe the reason that this is happening is because there is no check to see if a dagrun exists in `DagrunIdDep`: https://github.com/apache/airflow/blob/master/airflow/ti_deps/deps/dagrun_id_dep.py#L50 This is a bit of an edge case because there isn't really any reason someone should be checking out task instance details for a DAG that has never been run. That said we should probably avoid the error page. **How to reproduce it**: If you have a DAG that has not been turned on before and you click on that DAG in the admin page and then in the DAG graph click on one of the tasks and then click on `Task Instance Details` you get the "Ooops" explosion error page. **Anything else we need to know**: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] atsalolikhin-spokeo commented on issue #8388: Task instance details page blows up when no dagrun
atsalolikhin-spokeo commented on issue #8388: URL: https://github.com/apache/airflow/issues/8388#issuecomment-672214778 I'm sorry, I can't tell. I don't seem to have a fully functional Airflow installation. The console (CLI) keeps telling me: ``` Please make sure to build the frontend in static/ directory and restart the server ``` but I don't know how to do that. And the example bash operator disappeared from the UI after I did the "yarn prod run". This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #10280: Enable Sphinx spellcheck for docs generation
potiuk commented on pull request #10280: URL: https://github.com/apache/airflow/pull/10280#issuecomment-672125831 I re-run it. It looks like a transient error. Since we are modifying the dependencie pretty early in the Dockerfile, It gets rebuild from the point it is modified and sometimes there is a problem when downloading the packages where checksum does not match on some transient 404 not found error (usually this happens when apt get gets an error during download but this error is somehow swallowed by firewall or smth). And since we are rebuilding the image in all steps that need it, this might happen rather frequently. You will notice that in this PR each "Build CI image" takes much longer than usual 3-4 minutes (it might be 12 minutes or more because almost all image is rebuilt). BTW. This is one of the problems my (soon to submit) PR is going to address. We are going to build each image only once - so even if Dockerfile is modified at the beginning the build it will be built once and such pre-built image will be used by all the subsequent steps. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] atsalolikhin-spokeo commented on issue #8388: Task instance details page blows up when no dagrun
atsalolikhin-spokeo commented on issue #8388: URL: https://github.com/apache/airflow/issues/8388#issuecomment-672150672 Hello, I just installed Airflow for the first time, following the [Quick Start](https://airflow.apache.org/docs/stable/start.html) guide. I ran the bash example operator (still following the guide): ``` # run your first task instance airflow run example_bash_operator runme_0 2015-01-01 ``` And then, when I went to see how the run went, I got the error page with the mushroom cloud and the error. Now, this is on Apache Airflow v1.10.11 (latest stable), which I see was released on July 10, 2020. And this pull request was merged on April 15th. And this issue is closed. But I'm running into the same broken behavior. Here is my error page: ``` Ooops. / ( () ) \___ /( ( ( ) _)) ) )\ (( ( )() ) ( ) ) ((/ ( _( ) ( _) ) ( () ) ) ( ( ( (_) ((( ) .((_ ) . )_ ( ( )( ( )) ) . ) ( ) ( ( ( ( ) ( _ ( _) ). ) . ) ) ( ) ( ( ( ) ( ) ( )) ) _)( ) ) ) ( ( ( \ ) ((_ ( ) ( ) ) ) ) )) ( ) ( ( ( ( (_ ( ) ( _) ) ( ) ) ) ( ( ( ( ( ) (_ ) ) ) _) ) _( ( ) (( ( )(( _) _) _(_ ( (_ ) (_((__(_(__(( ( ( | ) ) ) )_))__))_)___) ((__)\\||lll|l||/// \_)) ( /(/ ( ) ) )\ ) (( ( ( | | ) ) )\ ) ( /(| / ( )) ) ) )) ) ( ( _(|)_) ) ( ||\(|(|)|/|| ) (|(||(||)) ( //|/l|||)|\\ \ ) (/ / // /|//\\ \ \ \ _) --- Node: ip-172-20-2-227.us-west-2.compute.internal --- Traceback (most recent call last): File "/usr/local/lib64/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib64/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib64/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib64/python3.7/site-packages/flask/_compat.py", line 39, in reraise raise value File "/usr/local/lib64/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib64/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/home/airflow/.local/lib/python3.7/site-packages/flask_admin/base.py", line 69, in inner return self._run_view(f, *args, **kwargs) File "/home/airflow/.local/lib/python3.7/site-packages/flask_admin/base.py", line 368, in _run_view return fn(self, *args, **kwargs) File "/usr/local/lib/python3.7/site-packages/flask_login/utils.py", line 258, in decorated_view return func(*args, **kwargs) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www/utils.py", line 286, in wrapper return f(*args, **kwargs) File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www/views.py", line 1055, in task dep_context=dep_context)] File "/home/airflow/.local/lib/python3.7/site-packages/airflow/www/views.py", line 1053, in failed_dep_reasons = [(dep.dep_name, dep.reason) for dep in File "/home/airflow/.local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 683, in get_failed_dep_statuses dep_context): File "/home/airflow/.local/lib/python3.7/site-packages/airflow/ti_deps/deps/base_ti_dep.py", line 106, in get_dep_statuses for dep_status in self._get_dep_statuses(ti, session, dep_context): File "/home/airflow/.local/lib/python3.7/site-packages/airflow/ti_deps/deps/dagrun_id_dep.py", line 51, in _get_dep_statuses if not dagrun.run_id or not match(BackfillJob.ID_PREFIX + '.*', dagrun.run_id): AttributeError: 'NoneType' object has no attribute 'run_id' ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL
[GitHub] [airflow] atsalolikhin-spokeo commented on issue #8388: Task instance details page blows up when no dagrun
atsalolikhin-spokeo commented on issue #8388: URL: https://github.com/apache/airflow/issues/8388#issuecomment-672183000 Here is a screnshot, showing there is no toggle switch like there was in 1.10.11  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] atsalolikhin-spokeo edited a comment on issue #8388: Task instance details page blows up when no dagrun
atsalolikhin-spokeo edited a comment on issue #8388: URL: https://github.com/apache/airflow/issues/8388#issuecomment-672183000 Here is a screenshot, showing there is no toggle switch like there was in 1.10.11  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] atsalolikhin-spokeo commented on issue #8388: Task instance details page blows up when no dagrun
atsalolikhin-spokeo commented on issue #8388:
URL: https://github.com/apache/airflow/issues/8388#issuecomment-672180803
Thank you! I figured it out, actually.
```
Error: Could not locate a Flask application. You did not provide the
"FLASK_APP" environment variable, and a "wsgi.py" or "app.py" module was not
found in the current directory.
[airflow@ip-172-20-2-227 airflow]$ find . -name wsgi.py
[airflow@ip-172-20-2-227 airflow]$ find . -name app.py
./airflow/www/app.py
[airflow@ip-172-20-2-227 airflow]$ cd airflow/www
[airflow@ip-172-20-2-227 www]$ flask fab create-user
Role [Public]: Admin
Username: admin
User first name: admin
User last name: admin
Email: ad...@example.com
Password:
Repeat for confirmation:
[2020-08-11 18:34:09,566] {dagbag.py:409} INFO - Filling up the DagBag from
/home/airflow/airflow/dags
Please make sure to build the frontend in static/ directory and restart the
server
[2020-08-11 18:34:09,735] {manager.py:719} WARNING - No user yet created,
use flask fab command to do it.
User admin created.
[airflow@ip-172-20-2-227 www]$
```
(I did read the documentation section you linked to, thank you; just after I
already created an `admin` user and logged in.)
Don't laugh, but I can't figure out how to unpause the bash example
operator. Instead of a sliding toggle switch, there is a checkbox.
I installed airflow by running "pip3 install --user ." in the top directory
of the top of master.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] atsalolikhin-spokeo commented on issue #8388: Task instance details page blows up when no dagrun
atsalolikhin-spokeo commented on issue #8388: URL: https://github.com/apache/airflow/issues/8388#issuecomment-672201902 I've installed JS using yarn as per the linked instructions. Now Airflow is showing 18 DAGs (before it was 22). The bash example operator DAG is gone. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch constraints-1-10 updated: Updating constraints. GH run id:204255384
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-1-10 in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-1-10 by this push: new 04c58ae Updating constraints. GH run id:204255384 04c58ae is described below commit 04c58ae6ad92be90c8775d4c47f34f84e4428edb Author: Automated Github Actions commit AuthorDate: Tue Aug 11 20:21:44 2020 + Updating constraints. GH run id:204255384 This update in constraints is automatically committed by CI pushing reference 'refs/heads/v1-10-test' to apache/airflow with commit sha a281faa00627ebdbde3c947e2857ae3e60b9273e. All tests passed in this build so we determined we can push the updated constraints. See https://github.com/apache/airflow/blob/master/README.md#installing-from-pypi for details. --- constraints-3.5.txt | 1 + constraints-3.6.txt | 2 +- constraints-3.8.txt | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/constraints-3.5.txt b/constraints-3.5.txt index 441cd41..38877ff 100644 --- a/constraints-3.5.txt +++ b/constraints-3.5.txt @@ -150,6 +150,7 @@ imagesize==1.2.0 importlib-metadata==1.7.0 importlib-resources==3.0.0 inflection==0.5.0 +iniconfig==1.0.1 ipdb==0.13.3 ipython-genutils==0.2.0 ipython==7.9.0 diff --git a/constraints-3.6.txt b/constraints-3.6.txt index 78cbf28..49a473f 100644 --- a/constraints-3.6.txt +++ b/constraints-3.6.txt @@ -96,7 +96,7 @@ docker-pycreds==0.4.0 docker==3.7.3 docopt==0.6.2 docutils==0.16 -ecdsa==0.15 +ecdsa==0.14.1 elasticsearch-dsl==5.4.0 elasticsearch==5.5.3 email-validator==1.1.1 diff --git a/constraints-3.8.txt b/constraints-3.8.txt index bff0628..c7d89e5 100644 --- a/constraints-3.8.txt +++ b/constraints-3.8.txt @@ -95,7 +95,7 @@ docker-pycreds==0.4.0 docker==3.7.3 docopt==0.6.2 docutils==0.16 -ecdsa==0.14.1 +ecdsa==0.15 elasticsearch-dsl==5.4.0 elasticsearch==5.5.3 email-validator==1.1.1
[airflow] branch constraints-1-10 updated: Updating constraints. GH run id:204255384
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-1-10 in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-1-10 by this push: new 04c58ae Updating constraints. GH run id:204255384 04c58ae is described below commit 04c58ae6ad92be90c8775d4c47f34f84e4428edb Author: Automated Github Actions commit AuthorDate: Tue Aug 11 20:21:44 2020 + Updating constraints. GH run id:204255384 This update in constraints is automatically committed by CI pushing reference 'refs/heads/v1-10-test' to apache/airflow with commit sha a281faa00627ebdbde3c947e2857ae3e60b9273e. All tests passed in this build so we determined we can push the updated constraints. See https://github.com/apache/airflow/blob/master/README.md#installing-from-pypi for details. --- constraints-3.5.txt | 1 + constraints-3.6.txt | 2 +- constraints-3.8.txt | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/constraints-3.5.txt b/constraints-3.5.txt index 441cd41..38877ff 100644 --- a/constraints-3.5.txt +++ b/constraints-3.5.txt @@ -150,6 +150,7 @@ imagesize==1.2.0 importlib-metadata==1.7.0 importlib-resources==3.0.0 inflection==0.5.0 +iniconfig==1.0.1 ipdb==0.13.3 ipython-genutils==0.2.0 ipython==7.9.0 diff --git a/constraints-3.6.txt b/constraints-3.6.txt index 78cbf28..49a473f 100644 --- a/constraints-3.6.txt +++ b/constraints-3.6.txt @@ -96,7 +96,7 @@ docker-pycreds==0.4.0 docker==3.7.3 docopt==0.6.2 docutils==0.16 -ecdsa==0.15 +ecdsa==0.14.1 elasticsearch-dsl==5.4.0 elasticsearch==5.5.3 email-validator==1.1.1 diff --git a/constraints-3.8.txt b/constraints-3.8.txt index bff0628..c7d89e5 100644 --- a/constraints-3.8.txt +++ b/constraints-3.8.txt @@ -95,7 +95,7 @@ docker-pycreds==0.4.0 docker==3.7.3 docopt==0.6.2 docutils==0.16 -ecdsa==0.14.1 +ecdsa==0.15 elasticsearch-dsl==5.4.0 elasticsearch==5.5.3 email-validator==1.1.1
[GitHub] [airflow] jaketf opened a new issue #10293: Add Dataproc Job Sensor
jaketf opened a new issue #10293: URL: https://github.com/apache/airflow/issues/10293 **Description** The Dataproc hook now has separate methods for [submitting](https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/hooks/dataproc.py#L778) and [waiting for job completion](https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/hooks/dataproc.py#L705). We should improve this further by adding a method to simply poke the status of the job. Finally implement a `DataprocJobSensor` so we can use reschedule mode for jobs known to run for a long time. **Use case / motivation** For Long running dataproc jobs that are known to run for a long time (e.g. > 1 hr) we should not block a slot for the entire duration of the job. **Related Issues** N/A **Other notes ** @varundhussa is currently working on this for a customer with this use case. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (AIRFLOW-5071) Thousand os Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?
[
https://issues.apache.org/jira/browse/AIRFLOW-5071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175866#comment-17175866
]
Nikolay commented on AIRFLOW-5071:
--
I was observing this issue, when multiple parallel instances of dags were
running. I'm using KubernetesExecutor. Also not only sensors were failing
(PythonOperator for example).
I decreased max_active_runs property, increased min_file_process_interval and
after that I haven't seen this issue for a while.
> Thousand os Executor reports task instance X finished (success) although the
> task says its queued. Was the task killed externally?
> --
>
> Key: AIRFLOW-5071
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5071
> Project: Apache Airflow
> Issue Type: Bug
> Components: DAG, scheduler
>Affects Versions: 1.10.3
>Reporter: msempere
>Priority: Critical
> Fix For: 1.10.12
>
> Attachments: image-2020-01-27-18-10-29-124.png,
> image-2020-07-08-07-58-42-972.png
>
>
> I'm opening this issue because since I update to 1.10.3 I'm seeing thousands
> of daily messages like the following in the logs:
>
> ```
> {{__init__.py:1580}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> {{jobs.py:1484}} ERROR - Executor reports task instance 2019-07-29 00:00:00+00:00 [queued]> finished (success) although the task says
> its queued. Was the task killed externally?
> ```
> -And looks like this is triggering also thousand of daily emails because the
> flag to send email in case of failure is set to True.-
> I have Airflow setup to use Celery and Redis as a backend queue service.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [airflow] darwinyip closed issue #10275: Trigger rule for "always"
darwinyip closed issue #10275: URL: https://github.com/apache/airflow/issues/10275 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] atsalolikhin-spokeo commented on issue #8388: Task instance details page blows up when no dagrun
atsalolikhin-spokeo commented on issue #8388: URL: https://github.com/apache/airflow/issues/8388#issuecomment-672152435 I don't have the rights to re-open this issue. Is it enough for me to comment on it here? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #10280: Enable Sphinx spellcheck for docs generation
kaxil commented on a change in pull request #10280: URL: https://github.com/apache/airflow/pull/10280#discussion_r468781032 ## File path: docs/spelling_wordlist.txt ## @@ -0,0 +1,1366 @@ +Ack +Acyclic +Airbnb +AirflowException +Aizhamal +Alphasort +Analytics +Anand +Ansible +AppBuilder +Arg +Args +Async +Auth +Autoscale +Avro +Azkaban +Azri +Backend +Backends +Backfill +BackfillJobTest +Backfills +Banco +Bas +BaseClient +BaseOperator +BaseView +Beauchemin +Behaviour +Bigquery +Bigtable +Bitshift +Bluecore +Bolke +Bool +Booleans +Boto +BounceX +Boxel +Bq +Breguła +CLoud +CSRFProtect +Cancelled +Cassanda +Catchup +Celect +Cgroups +Changelog +Chao +CheckOperator +Checklicence +Checkr +Cinimex +Cloudant +Cloudwatch +ClusterManagerClient +Codecov +Colour +ComputeNodeState +Computenodes +Config +Configs +Cron +Ctrl +Daemonize +DagFileProcessorManager +DagRun +Dagbag +Dagre +Dask +Databricks +Datadog +Dataflow +Dataframe +Datalake +Datanodes +Dataprep +Dataproc +Dataset +Datasets +Datastore +Datasync +DateFrame +Datetimepicker +Datetimes +Davydov +De +Decrypt +Decrypts +Deserialize +Deserialized +Dingding +Dlp +Dockerfile +Dockerhub +Dockerise +Docstring +Docstrings +Dont +Driesprong +Drivy +Dsn +Dynamodb +EDITMSG +ETag +Eg +EmrAddSteps +EmrCreateJobFlow +Enum +Env +Exasol +Failover +Feng +Fernet +FileSensor +Filebeat +Filepath +Fileshare +Fileshares +Filesystem +Firehose +Firestore +Fokko +Formaturas +Fundera +GCP +GCS +GH +GSoD +Gannt +Gantt +Gao +Gcp +Gentner +GetPartitions +GiB +Github +Gitter +Glassdoor +Gmail +Groupalia +Groupon +Grpc +Gunicorn +Guziel +Gzip +HCatalog +HTTPBasicAuth +Harenslak +Hashable +Hashicorp +Highcharts +Hitesh +Hiveserver +Hoc +Homan +Hostname +Hou +Http +HttpError +HttpRequest +IdP +Imap +Imberman +InsecureClient +InspectContentResponse +Investorise +JPype +Jakob +Jarek +Jdbc +Jiajie +Jinja +Jinjafied +Jinjafy +Jira +JobComplete +JobExists +JobRunning +Json +Jupyter +KYLIN +Kalibrr +Kamil +Kaxil +Kengo +Kerberized +Kerberos +KerberosClient +KevinYang +KeyManagementServiceClient +Keyfile +Kibana +Kinesis +Kombu +Kube +Kubernetes +Kusto +Kwargs +Kylin +LaunchTemplateParameters +Letsbonus +Lifecycle +LineItem +ListGenerator +Logstash +Lowin +Lyft +Maheshwari +Makefile +Mapreduce +Masternode +Maxime +Memorystore +Mesos +MessageAttributes +Metastore +Metatsore +Mixin +Mongo +Moto +Mysql +NFS +NaN +Naik +Namenode +Namespace +Nextdoor +Nones +NotFound +Nullable +Nurmamat +OAuth +Oauth +Oauthlib +Okta +Oozie +Opsgenie +Optimise +PEM +PTarget +Pagerduty +Papermill +Parallelize +Parameterizing +Paramiko +Params +Pem +Pinot +Popen +Postgres +Postgresql +Potiuk +Pre +Precommit +Preprocessed +Proc +Protobuf +PublisherClient +Pubsub +Py +PyPI +Pylint +Pyspark +PythonOperator +Qingping +Qplum +Quantopian +Qubole +Quboles +Quoble +RBAC +Readme +Realtime +Rebasing +Rebrand +Reddit +Redhat +Reinitialising +Retrives +Riccomini +Roadmap +Robinhood +SIGTERM +SSHClient +SSHTunnelForwarder +SaaS +Sagemaker +Sasl +SecretManagerClient +Seedlist +Seki +Sendgrid +Siddharth +SlackHook +SparkPi +SparkR +SparkSQL +Sql +Sqlalchemy +Sqlite +Sqoop +Stackdriver +Standarization +StatsD +Statsd +Stringified +Subclasses +Subdirectory +Submodules +Subpackages +Subpath +SubscriberClient +Subtasks +Sumit +Swtch +Systemd +TCP +TLS +TTY +TThe +TZ +TaskInstance +Taskfail +Templated +Templating +Teradata +TextToSpeechClient +Tez +Thinknear +ToC +Tomasz +Tooltip +Tsai +UA +UNload +Uellendall +Umask +Un +Undead +Undeads +Unpausing +Unpickle +Upsert +Upsight +Uptime +Urbaszek +Url +Utils +Vendorize +Vertica +Vevo +WTF +WaiterModel +Wasb +WebClient +Webhook +Webserver +Werkzeug +Wiedmer +XCom +XComs +Xcom +Xero +Xiaodong +Yamllint +Yandex +Yieldr +Zego +Zendesk +Zhong +Zsh +Zymergen +aIRFLOW +abc +accessor +accountmaking +acessible +ack +ackIds +acknowledgement +actionCard +acurate +acyclic +adhoc +aijamalnk +airbnb +airfl +airflowignore +ajax +alertPolicies +alexvanboxel +allAuthenticatedUsers +allUsers +allowinsert +analyse +analytics +analyticsreporting +analyzeEntities +analyzeSentiment +analyzeSyntax +aoen +apache +api +apikey +apis +appbuilder +approle +arg +args +arn +arraysize +artwr +asc +ascii +asciiart +asia +assertEqualIgnoreMultipleSpaces +ast +async +athena +attemping +attr +attrs +auth +authMechanism +authenication +authenticator +authorised +autoclass +autocommit +autocomplete +autodetect +autodetected +autoenv +autogenerated +autorestart +autoscale +autoscaling +avro +aws +awsbatch +backend +backends +backfill +backfilled +backfilling +backfills +backoff +backport +backreference +backtick +backticks +balancer +baseOperator +basedn +basestring +basetaskrunner +bashrc +basph +batchGet +bc +bcc +beeen +behaviour +behaviours +behavoiur +bicket +bigquery +bigtable +bitshift +bolkedebruin +booktabs +boolean +booleans +bootDiskType +boto +botocore +bowser +bq +bugfix +bugfixes +buildType +bytestring +cacert +callables +cancelled +carbonite +cas +cassandra +casted +catchup +cattrs +ccache +celeryd
[GitHub] [airflow] atsalolikhin-spokeo edited a comment on issue #8388: Task instance details page blows up when no dagrun
atsalolikhin-spokeo edited a comment on issue #8388:
URL: https://github.com/apache/airflow/issues/8388#issuecomment-672169136
Thank you for re-opening this issue. I've installed the master version:
```
[airflow@ip-172-20-2-227 airflow]$ airflow version
2.0.0.dev0
[airflow@ip-172-20-2-227 airflow]$
```
I can't login through the Web UI, though. It's asking me for username and
password.
When I started the webserver, it said:
[2020-08-11 18:25:27,417] {manager.py:719} WARNING - No user yet created,
use flask fab command to do it.
I tried, but it didn't work. Help?
```
[airflow@ip-172-20-2-227 airflow]$ flask fab create-user
Role [Public]:
Username: admin
User first name: admin
User last name: admin
Email: ad...@example.com
Password:
Repeat for confirmation:
Usage: flask fab create-user [OPTIONS]
Error: Could not locate a Flask application. You did not provide the
"FLASK_APP" environment variable, and a "wsgi.py" or "app.py" module was not
found in the current directory.
[airflow@ip-172-20-2-227 airflow]$
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #10227: Use Hash of Serialized DAG to determine DAG is changed or not
kaxil commented on a change in pull request #10227:
URL: https://github.com/apache/airflow/pull/10227#discussion_r468781710
##
File path: airflow/models/serialized_dag.py
##
@@ -102,9 +109,11 @@ def write_dag(cls, dag: DAG, min_update_interval:
Optional[int] = None, session:
return
log.debug("Checking if DAG (%s) changed", dag.dag_id)
-serialized_dag_from_db: SerializedDagModel =
session.query(cls).get(dag.dag_id)
new_serialized_dag = cls(dag)
-if serialized_dag_from_db and (serialized_dag_from_db.data ==
new_serialized_dag.data):
+serialized_dag_hash_from_db = session.query(
+cls.dag_hash).filter(cls.dag_id == dag.dag_id).scalar()
+
+if serialized_dag_hash_from_db and (serialized_dag_hash_from_db ==
new_serialized_dag.dag_hash):
Review comment:
Updated
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [airflow] atsalolikhin-spokeo commented on issue #8388: Task instance details page blows up when no dagrun
atsalolikhin-spokeo commented on issue #8388: URL: https://github.com/apache/airflow/issues/8388#issuecomment-672194980 Okay. I used `sqlite3` to change the setting in `airflow.db` via CLI as you suggested. Now the UI is showing all 22 DAGs are unpaused. ```sql update dag set is_paused = '0'; ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #8388: Task instance details page blows up when no dagrun
mik-laj edited a comment on issue #8388: URL: https://github.com/apache/airflow/issues/8388#issuecomment-672186288 You should compile JS files now, but this is a problematic process. https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#installing-yarn-and-its-packages Another solution: can you try to unpause the DAG with CLI and then go directly to the URL? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] kaxil closed issue #10116: Use Hash / SHA to compare Serialized DAGs
kaxil closed issue #10116: URL: https://github.com/apache/airflow/issues/10116 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] kaxil merged pull request #10227: Use Hash of Serialized DAG to determine DAG is changed or not
kaxil merged pull request #10227: URL: https://github.com/apache/airflow/pull/10227 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] branch master updated: Use Hash of Serialized DAG to determine DAG is changed or not (#10227)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/master by this push:
new adce6f0 Use Hash of Serialized DAG to determine DAG is changed or not
(#10227)
adce6f0 is described below
commit adce6f029609e89f3651a89df40700589ec16237
Author: Kaxil Naik
AuthorDate: Tue Aug 11 22:31:55 2020 +0100
Use Hash of Serialized DAG to determine DAG is changed or not (#10227)
closes #10116
---
...c3a5a_add_dag_hash_column_to_serialized_dag_.py | 46 ++
airflow/models/serialized_dag.py | 11 --
tests/models/test_serialized_dag.py| 17
3 files changed, 62 insertions(+), 12 deletions(-)
diff --git
a/airflow/migrations/versions/da3f683c3a5a_add_dag_hash_column_to_serialized_dag_.py
b/airflow/migrations/versions/da3f683c3a5a_add_dag_hash_column_to_serialized_dag_.py
new file mode 100644
index 000..4dbc77a
--- /dev/null
+++
b/airflow/migrations/versions/da3f683c3a5a_add_dag_hash_column_to_serialized_dag_.py
@@ -0,0 +1,46 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Add dag_hash Column to serialized_dag table
+
+Revision ID: da3f683c3a5a
+Revises: 8d48763f6d53
+Create Date: 2020-08-07 20:52:09.178296
+
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = 'da3f683c3a5a'
+down_revision = '8d48763f6d53'
+branch_labels = None
+depends_on = None
+
+
+def upgrade():
+"""Apply Add dag_hash Column to serialized_dag table"""
+op.add_column(
+'serialized_dag',
+sa.Column('dag_hash', sa.String(32), nullable=False,
server_default='Hash not calculated yet'))
+
+
+def downgrade():
+"""Unapply Add dag_hash Column to serialized_dag table"""
+op.drop_column('serialized_dag', 'dag_hash')
diff --git a/airflow/models/serialized_dag.py b/airflow/models/serialized_dag.py
index 078724d..2368b9e 100644
--- a/airflow/models/serialized_dag.py
+++ b/airflow/models/serialized_dag.py
@@ -18,6 +18,7 @@
"""Serialized DAG table in database."""
+import hashlib
import logging
from datetime import datetime, timedelta
from typing import Any, Dict, List, Optional
@@ -53,7 +54,7 @@ class SerializedDagModel(Base):
interval of deleting serialized DAGs in DB when the files are deleted,
suggest
to use a smaller interval such as 60
-It is used by webserver to load dagbags when
``store_serialized_dags=True``.
+It is used by webserver to load dags when ``store_serialized_dags=True``.
Because reading from database is lightweight compared to importing from
files,
it solves the webserver scalability issue.
"""
@@ -65,6 +66,7 @@ class SerializedDagModel(Base):
fileloc_hash = Column(BigInteger, nullable=False)
data = Column(sqlalchemy_jsonfield.JSONField(json=json), nullable=False)
last_updated = Column(UtcDateTime, nullable=False)
+dag_hash = Column(String(32), nullable=False)
__table_args__ = (
Index('idx_fileloc_hash', fileloc_hash, unique=False),
@@ -76,6 +78,7 @@ class SerializedDagModel(Base):
self.fileloc_hash = DagCode.dag_fileloc_hash(self.fileloc)
self.data = SerializedDAG.to_dict(dag)
self.last_updated = timezone.utcnow()
+self.dag_hash = hashlib.md5(json.dumps(self.data,
sort_keys=True).encode("utf-8")).hexdigest()
def __repr__(self):
return f""
@@ -102,9 +105,11 @@ class SerializedDagModel(Base):
return
log.debug("Checking if DAG (%s) changed", dag.dag_id)
-serialized_dag_from_db: SerializedDagModel =
session.query(cls).get(dag.dag_id)
new_serialized_dag = cls(dag)
-if serialized_dag_from_db and (serialized_dag_from_db.data ==
new_serialized_dag.data):
+serialized_dag_hash_from_db = session.query(
+cls.dag_hash).filter(cls.dag_id == dag.dag_id).scalar()
+
+if serialized_dag_hash_from_db == new_serialized_dag.dag_hash:
log.debug("Serialized DAG (%s) is unchanged. Skipping writing to
DB", dag.dag_id)
return
[GitHub] [airflow] kaxil commented on issue #10264: Add Sphinx Spell-Checker
kaxil commented on issue #10264: URL: https://github.com/apache/airflow/issues/10264#issuecomment-672108330 > 1. Should we have a specific command to run spell check other than building the docs? in breeze too? I took a look at your PR and I think running the spell check as part of the doc only makes sense for now. We don't need a separate command for it. Even if we would want to do it, it can happen in a separate PR :) > 2. Should I my PR to update the spellings for as many of the misspellings as I can? It will boost the number of files I change, obviously. Yes, otherwise the PR would fail so atleast I would expect it to fail. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] rootcss commented on issue #8715: Documentation about loading packages in Python / Airflow
rootcss commented on issue #8715: URL: https://github.com/apache/airflow/issues/8715#issuecomment-672108560 @mik-laj where do you think we should put this doc? a new file in the root of docs or somewhere else? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (AIRFLOW-3964) Consolidate and de-duplicate sensor tasks
[
https://issues.apache.org/jira/browse/AIRFLOW-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175743#comment-17175743
]
ASF GitHub Bot commented on AIRFLOW-3964:
-
YingboWang commented on a change in pull request #5499:
URL: https://github.com/apache/airflow/pull/5499#discussion_r468753067
##
File path: airflow/sensors/smart_sensor_operator.py
##
@@ -0,0 +1,730 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import datetime
+import json
+import logging
+import time
+import traceback
+from logging.config import DictConfigurator # type: ignore
+from time import sleep
+
+from sqlalchemy import and_, or_, tuple_
+
+from airflow.exceptions import AirflowException, AirflowTaskTimeout
+from airflow.models import BaseOperator, SensorInstance, SkipMixin,
TaskInstance
+from airflow.settings import LOGGING_CLASS_PATH
+from airflow.stats import Stats
+from airflow.utils import helpers, timezone
+from airflow.utils.decorators import apply_defaults
+from airflow.utils.email import send_email
+from airflow.utils.log.logging_mixin import set_context
+from airflow.utils.module_loading import import_string
+from airflow.utils.net import get_hostname
+from airflow.utils.session import provide_session
+from airflow.utils.state import PokeState, State
+from airflow.utils.timeout import timeout
+
+config = import_string(LOGGING_CLASS_PATH)
+handler_config = config['handlers']['task']
+try:
+formatter_config = config['formatters'][handler_config['formatter']]
+except Exception as err: # pylint: disable=broad-except
+formatter_config = None
+print(err)
+dictConfigurator = DictConfigurator(config)
+
+
+class SensorWork:
+"""
+This class stores a sensor work with decoded context value. It is only used
+inside of smart sensor.
+"""
+def __init__(self, ti):
+self.dag_id = ti.dag_id
+self.task_id = ti.task_id
+self.execution_date = ti.execution_date
+self.try_number = ti.try_number
+
+self.poke_context = json.loads(ti.poke_context) if ti.poke_context
else {}
+self.execution_context = json.loads(ti.execution_context) if
ti.execution_context else {}
+try:
+self.log = self._get_sensor_logger(ti)
+except Exception as e: # pylint: disable=broad-except
+self.log = None
+print(e)
+self.hashcode = ti.hashcode
+self.start_date = ti.start_date
+self.operator = ti.operator
+self.op_classpath = ti.op_classpath
+self.encoded_poke_context = ti.poke_context
+
+def __eq__(self, other):
+if not isinstance(other, SensorWork):
+return NotImplemented
+
+return self.dag_id == other.dag_id and \
+self.task_id == other.task_id and \
+self.execution_date == other.execution_date and \
+self.try_number == other.try_number
+
+@staticmethod
+def create_new_task_handler():
+"""
+Create task log handler for a sensor work.
+:return: log handler
+"""
+handler_config_copy = {k: handler_config[k] for k in handler_config}
+formatter_config_copy = {k: formatter_config[k] for k in
formatter_config}
+handler = dictConfigurator.configure_handler(handler_config_copy)
+formatter = dictConfigurator.configure_formatter(formatter_config_copy)
+handler.setFormatter(formatter)
+return handler
+
+def _get_sensor_logger(self, ti):
+# TODO: should be somewhere else, but not this file, has to use
LOG_ID_TEMPLATE from es
+# but how about other log file handler?
+ti.raw = False # Otherwise set_context will fail
+log_id = "-".join([ti.dag_id,
+ ti.task_id,
+ ti.execution_date.strftime("%Y_%m_%dT%H_%M_%S_%f"),
+ str(ti.try_number)])
+logger = logging.getLogger('airflow.task' + '.' + log_id)
+
+if len(logger.handlers) == 0:
+handler = self.create_new_task_handler()
+logger.addHandler(handler)
+set_context(logger, ti)
+
+line_break =
[GitHub] [airflow] YingboWang commented on a change in pull request #5499: [AIRFLOW-3964][AIP-17] Build smart sensor
YingboWang commented on a change in pull request #5499:
URL: https://github.com/apache/airflow/pull/5499#discussion_r468753067
##
File path: airflow/sensors/smart_sensor_operator.py
##
@@ -0,0 +1,730 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import datetime
+import json
+import logging
+import time
+import traceback
+from logging.config import DictConfigurator # type: ignore
+from time import sleep
+
+from sqlalchemy import and_, or_, tuple_
+
+from airflow.exceptions import AirflowException, AirflowTaskTimeout
+from airflow.models import BaseOperator, SensorInstance, SkipMixin,
TaskInstance
+from airflow.settings import LOGGING_CLASS_PATH
+from airflow.stats import Stats
+from airflow.utils import helpers, timezone
+from airflow.utils.decorators import apply_defaults
+from airflow.utils.email import send_email
+from airflow.utils.log.logging_mixin import set_context
+from airflow.utils.module_loading import import_string
+from airflow.utils.net import get_hostname
+from airflow.utils.session import provide_session
+from airflow.utils.state import PokeState, State
+from airflow.utils.timeout import timeout
+
+config = import_string(LOGGING_CLASS_PATH)
+handler_config = config['handlers']['task']
+try:
+formatter_config = config['formatters'][handler_config['formatter']]
+except Exception as err: # pylint: disable=broad-except
+formatter_config = None
+print(err)
+dictConfigurator = DictConfigurator(config)
+
+
+class SensorWork:
+"""
+This class stores a sensor work with decoded context value. It is only used
+inside of smart sensor.
+"""
+def __init__(self, ti):
+self.dag_id = ti.dag_id
+self.task_id = ti.task_id
+self.execution_date = ti.execution_date
+self.try_number = ti.try_number
+
+self.poke_context = json.loads(ti.poke_context) if ti.poke_context
else {}
+self.execution_context = json.loads(ti.execution_context) if
ti.execution_context else {}
+try:
+self.log = self._get_sensor_logger(ti)
+except Exception as e: # pylint: disable=broad-except
+self.log = None
+print(e)
+self.hashcode = ti.hashcode
+self.start_date = ti.start_date
+self.operator = ti.operator
+self.op_classpath = ti.op_classpath
+self.encoded_poke_context = ti.poke_context
+
+def __eq__(self, other):
+if not isinstance(other, SensorWork):
+return NotImplemented
+
+return self.dag_id == other.dag_id and \
+self.task_id == other.task_id and \
+self.execution_date == other.execution_date and \
+self.try_number == other.try_number
+
+@staticmethod
+def create_new_task_handler():
+"""
+Create task log handler for a sensor work.
+:return: log handler
+"""
+handler_config_copy = {k: handler_config[k] for k in handler_config}
+formatter_config_copy = {k: formatter_config[k] for k in
formatter_config}
+handler = dictConfigurator.configure_handler(handler_config_copy)
+formatter = dictConfigurator.configure_formatter(formatter_config_copy)
+handler.setFormatter(formatter)
+return handler
+
+def _get_sensor_logger(self, ti):
+# TODO: should be somewhere else, but not this file, has to use
LOG_ID_TEMPLATE from es
+# but how about other log file handler?
+ti.raw = False # Otherwise set_context will fail
+log_id = "-".join([ti.dag_id,
+ ti.task_id,
+ ti.execution_date.strftime("%Y_%m_%dT%H_%M_%S_%f"),
+ str(ti.try_number)])
+logger = logging.getLogger('airflow.task' + '.' + log_id)
+
+if len(logger.handlers) == 0:
+handler = self.create_new_task_handler()
+logger.addHandler(handler)
+set_context(logger, ti)
+
+line_break = ("-" * 120)
+logger.info(line_break)
+logger.info("Processing sensor task %s in smart sensor service on
host: %s",
+self.ti_key, get_hostname())
+logger.info(line_break)
+
[GitHub] [airflow] mik-laj commented on issue #8715: Documentation about loading packages in Python / Airflow
mik-laj commented on issue #8715: URL: https://github.com/apache/airflow/issues/8715#issuecomment-672147126 New file in the root directory - `docs/` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #8715: Documentation about loading packages in Python / Airflow
mik-laj edited a comment on issue #8715: URL: https://github.com/apache/airflow/issues/8715#issuecomment-672147126 The new file in the root directory - `docs/` sounds good to me. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #8388: Task instance details page blows up when no dagrun
mik-laj commented on issue #8388: URL: https://github.com/apache/airflow/issues/8388#issuecomment-672172264 Here is docs about your problem: https://github.com/apache/airflow/blob/master/UPDATING.md#drop-legacy-ui-in-favor-of-fab-rbac-ui This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (AIRFLOW-3964) Consolidate and de-duplicate sensor tasks
[
https://issues.apache.org/jira/browse/AIRFLOW-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175780#comment-17175780
]
ASF GitHub Bot commented on AIRFLOW-3964:
-
YingboWang commented on a change in pull request #5499:
URL: https://github.com/apache/airflow/pull/5499#discussion_r468792189
##
File path: airflow/sensors/smart_sensor_operator.py
##
@@ -0,0 +1,730 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import datetime
+import json
+import logging
+import time
+import traceback
+from logging.config import DictConfigurator # type: ignore
+from time import sleep
+
+from sqlalchemy import and_, or_, tuple_
+
+from airflow.exceptions import AirflowException, AirflowTaskTimeout
+from airflow.models import BaseOperator, SensorInstance, SkipMixin,
TaskInstance
+from airflow.settings import LOGGING_CLASS_PATH
+from airflow.stats import Stats
+from airflow.utils import helpers, timezone
+from airflow.utils.decorators import apply_defaults
+from airflow.utils.email import send_email
+from airflow.utils.log.logging_mixin import set_context
+from airflow.utils.module_loading import import_string
+from airflow.utils.net import get_hostname
+from airflow.utils.session import provide_session
+from airflow.utils.state import PokeState, State
+from airflow.utils.timeout import timeout
+
+config = import_string(LOGGING_CLASS_PATH)
+handler_config = config['handlers']['task']
+try:
+formatter_config = config['formatters'][handler_config['formatter']]
+except Exception as err: # pylint: disable=broad-except
+formatter_config = None
+print(err)
+dictConfigurator = DictConfigurator(config)
+
+
+class SensorWork:
+"""
+This class stores a sensor work with decoded context value. It is only used
+inside of smart sensor.
+"""
+def __init__(self, ti):
+self.dag_id = ti.dag_id
+self.task_id = ti.task_id
+self.execution_date = ti.execution_date
+self.try_number = ti.try_number
+
+self.poke_context = json.loads(ti.poke_context) if ti.poke_context
else {}
+self.execution_context = json.loads(ti.execution_context) if
ti.execution_context else {}
+try:
+self.log = self._get_sensor_logger(ti)
+except Exception as e: # pylint: disable=broad-except
+self.log = None
+print(e)
+self.hashcode = ti.hashcode
+self.start_date = ti.start_date
+self.operator = ti.operator
+self.op_classpath = ti.op_classpath
+self.encoded_poke_context = ti.poke_context
+
+def __eq__(self, other):
+if not isinstance(other, SensorWork):
+return NotImplemented
+
+return self.dag_id == other.dag_id and \
+self.task_id == other.task_id and \
+self.execution_date == other.execution_date and \
+self.try_number == other.try_number
+
+@staticmethod
+def create_new_task_handler():
+"""
+Create task log handler for a sensor work.
+:return: log handler
+"""
+handler_config_copy = {k: handler_config[k] for k in handler_config}
+formatter_config_copy = {k: formatter_config[k] for k in
formatter_config}
+handler = dictConfigurator.configure_handler(handler_config_copy)
+formatter = dictConfigurator.configure_formatter(formatter_config_copy)
+handler.setFormatter(formatter)
+return handler
+
+def _get_sensor_logger(self, ti):
+# TODO: should be somewhere else, but not this file, has to use
LOG_ID_TEMPLATE from es
+# but how about other log file handler?
+ti.raw = False # Otherwise set_context will fail
+log_id = "-".join([ti.dag_id,
+ ti.task_id,
+ ti.execution_date.strftime("%Y_%m_%dT%H_%M_%S_%f"),
Review comment:
Should not fail. The log_id is used as an internal key for fetching
these in memory log handler only in smart sensor service. The output path was
configure in the `set_context` function which should always be consistent with
other tasks.
[GitHub] [airflow] YingboWang commented on a change in pull request #5499: [AIRFLOW-3964][AIP-17] Build smart sensor
YingboWang commented on a change in pull request #5499:
URL: https://github.com/apache/airflow/pull/5499#discussion_r468792189
##
File path: airflow/sensors/smart_sensor_operator.py
##
@@ -0,0 +1,730 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+import datetime
+import json
+import logging
+import time
+import traceback
+from logging.config import DictConfigurator # type: ignore
+from time import sleep
+
+from sqlalchemy import and_, or_, tuple_
+
+from airflow.exceptions import AirflowException, AirflowTaskTimeout
+from airflow.models import BaseOperator, SensorInstance, SkipMixin,
TaskInstance
+from airflow.settings import LOGGING_CLASS_PATH
+from airflow.stats import Stats
+from airflow.utils import helpers, timezone
+from airflow.utils.decorators import apply_defaults
+from airflow.utils.email import send_email
+from airflow.utils.log.logging_mixin import set_context
+from airflow.utils.module_loading import import_string
+from airflow.utils.net import get_hostname
+from airflow.utils.session import provide_session
+from airflow.utils.state import PokeState, State
+from airflow.utils.timeout import timeout
+
+config = import_string(LOGGING_CLASS_PATH)
+handler_config = config['handlers']['task']
+try:
+formatter_config = config['formatters'][handler_config['formatter']]
+except Exception as err: # pylint: disable=broad-except
+formatter_config = None
+print(err)
+dictConfigurator = DictConfigurator(config)
+
+
+class SensorWork:
+"""
+This class stores a sensor work with decoded context value. It is only used
+inside of smart sensor.
+"""
+def __init__(self, ti):
+self.dag_id = ti.dag_id
+self.task_id = ti.task_id
+self.execution_date = ti.execution_date
+self.try_number = ti.try_number
+
+self.poke_context = json.loads(ti.poke_context) if ti.poke_context
else {}
+self.execution_context = json.loads(ti.execution_context) if
ti.execution_context else {}
+try:
+self.log = self._get_sensor_logger(ti)
+except Exception as e: # pylint: disable=broad-except
+self.log = None
+print(e)
+self.hashcode = ti.hashcode
+self.start_date = ti.start_date
+self.operator = ti.operator
+self.op_classpath = ti.op_classpath
+self.encoded_poke_context = ti.poke_context
+
+def __eq__(self, other):
+if not isinstance(other, SensorWork):
+return NotImplemented
+
+return self.dag_id == other.dag_id and \
+self.task_id == other.task_id and \
+self.execution_date == other.execution_date and \
+self.try_number == other.try_number
+
+@staticmethod
+def create_new_task_handler():
+"""
+Create task log handler for a sensor work.
+:return: log handler
+"""
+handler_config_copy = {k: handler_config[k] for k in handler_config}
+formatter_config_copy = {k: formatter_config[k] for k in
formatter_config}
+handler = dictConfigurator.configure_handler(handler_config_copy)
+formatter = dictConfigurator.configure_formatter(formatter_config_copy)
+handler.setFormatter(formatter)
+return handler
+
+def _get_sensor_logger(self, ti):
+# TODO: should be somewhere else, but not this file, has to use
LOG_ID_TEMPLATE from es
+# but how about other log file handler?
+ti.raw = False # Otherwise set_context will fail
+log_id = "-".join([ti.dag_id,
+ ti.task_id,
+ ti.execution_date.strftime("%Y_%m_%dT%H_%M_%S_%f"),
Review comment:
Should not fail. The log_id is used as an internal key for fetching
these in memory log handler only in smart sensor service. The output path was
configure in the `set_context` function which should always be consistent with
other tasks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure
[GitHub] [airflow] mik-laj commented on issue #8388: Task instance details page blows up when no dagrun
mik-laj commented on issue #8388: URL: https://github.com/apache/airflow/issues/8388#issuecomment-672186288 You should now compile JS files, but this is a problematic process. https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#installing-yarn-and-its-packages Other solution: can you try to unpause the DAG with CLI and then go directly to the URL? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] atsalolikhin-spokeo commented on issue #8388: Task instance details page blows up when no dagrun
atsalolikhin-spokeo commented on issue #8388: URL: https://github.com/apache/airflow/issues/8388#issuecomment-672217687 I triggered the "tutorial" DAG instead (from the UI), and it looks like it ran. I can look at the task details. No mushroom cloud. So I would say this issue is fixed in master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #10267: Add Authentication for Stable API
ashb commented on pull request #10267: URL: https://github.com/apache/airflow/pull/10267#issuecomment-672223632 Does this do everything we need for https://github.com/apache/airflow/issues/8111? Additionally why did you do this PR without first checking in with James when that ticket was assigned to him? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[airflow] 07/32: Remove package.json and yarn.lock from the prod image (#9814)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 1d4782e4e2061d8b8368afc698ef97ec67ca360c
Author: Jarek Potiuk
AuthorDate: Tue Jul 14 16:34:21 2020 +0200
Remove package.json and yarn.lock from the prod image (#9814)
Closes #9810
(cherry picked from commit 593a0ddaae2deaa283c260a32187cf3c27ec3e7d)
---
Dockerfile | 1 +
1 file changed, 1 insertion(+)
diff --git a/Dockerfile b/Dockerfile
index a882178..c06105d 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -225,6 +225,7 @@ RUN
AIRFLOW_SITE_PACKAGE="/root/.local/lib/python${PYTHON_MAJOR_MINOR_VERSION}/s
yarn --cwd "${WWW_DIR}" install --frozen-lockfile --no-cache; \
yarn --cwd "${WWW_DIR}" run prod; \
rm -rf "${WWW_DIR}/node_modules"; \
+rm -vf
"${WWW_DIR}"/{package.json,yarn.lock,.eslintignore,.eslintrc,.stylelintignore,.stylelintrc,compile_assets.sh,webpack.config.js}
;\
fi
# make sure that all directories and files in .local are also group accessible
[airflow] 31/32: Fix KubernetesPodOperator reattachment (#10230)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit c47a7c443056382401c05363d0e57b8301f1bf31
Author: Daniel Imberman
AuthorDate: Tue Aug 11 07:01:27 2020 -0700
Fix KubernetesPodOperator reattachment (#10230)
(cherry picked from commit 8cd2be9e161635480581a0dc723b69ed24166f8d)
---
.../contrib/operators/kubernetes_pod_operator.py | 46 --
1 file changed, 33 insertions(+), 13 deletions(-)
diff --git a/airflow/contrib/operators/kubernetes_pod_operator.py
b/airflow/contrib/operators/kubernetes_pod_operator.py
index 41f0df3..98464b7 100644
--- a/airflow/contrib/operators/kubernetes_pod_operator.py
+++ b/airflow/contrib/operators/kubernetes_pod_operator.py
@@ -270,23 +270,16 @@ class KubernetesPodOperator(BaseOperator): # pylint:
disable=too-many-instance-
pod_list = client.list_namespaced_pod(self.namespace,
label_selector=label_selector)
-if len(pod_list.items) > 1:
+if len(pod_list.items) > 1 and self.reattach_on_restart:
raise AirflowException(
'More than one pod running with labels: '
'{label_selector}'.format(label_selector=label_selector))
launcher = pod_launcher.PodLauncher(kube_client=client,
extract_xcom=self.do_xcom_push)
-if len(pod_list.items) == 1 and \
-self._try_numbers_do_not_match(context, pod_list.items[0])
and \
-self.reattach_on_restart:
-self.log.info("found a running pod with labels %s but a
different try_number"
- "Will attach to this pod and monitor instead of
starting new one", labels)
-final_state, _, result =
self.create_new_pod_for_operator(labels, launcher)
-elif len(pod_list.items) == 1:
-self.log.info("found a running pod with labels %s."
- "Will monitor this pod instead of starting new
one", labels)
-final_state, result = self.monitor_launched_pod(launcher,
pod_list[0])
+if len(pod_list.items) == 1:
+try_numbers_match = self._try_numbers_match(context,
pod_list.items[0])
+final_state, result = self.handle_pod_overlap(labels,
try_numbers_match, launcher, pod_list)
else:
final_state, _, result =
self.create_new_pod_for_operator(labels, launcher)
if final_state != State.SUCCESS:
@@ -296,14 +289,41 @@ class KubernetesPodOperator(BaseOperator): # pylint:
disable=too-many-instance-
except AirflowException as ex:
raise AirflowException('Pod Launching failed:
{error}'.format(error=ex))
+def handle_pod_overlap(self, labels, try_numbers_match, launcher,
pod_list):
+"""
+In cases where the Scheduler restarts while a KubernetsPodOperator
task is running,
+this function will either continue to monitor the existing pod or
launch a new pod
+based on the `reattach_on_restart` parameter.
+:param labels: labels used to determine if a pod is repeated
+:type labels: dict
+:param try_numbers_match: do the try numbers match? Only needed for
logging purposes
+:type try_numbers_match: bool
+:param launcher: PodLauncher
+:param pod_list: list of pods found
+"""
+if try_numbers_match:
+log_line = "found a running pod with labels {} and the same
try_number.".format(labels)
+else:
+log_line = "found a running pod with labels {} but a different
try_number.".format(labels)
+
+if self.reattach_on_restart:
+log_line = log_line + " Will attach to this pod and monitor
instead of starting new one"
+self.log.info(log_line)
+final_state, result = self.monitor_launched_pod(launcher,
pod_list.items[0])
+else:
+log_line = log_line + "creating pod with labels {} and launcher
{}".format(labels, launcher)
+self.log.info(log_line)
+final_state, _, result = self.create_new_pod_for_operator(labels,
launcher)
+return final_state, result
+
@staticmethod
def _get_pod_identifying_label_string(labels):
filtered_labels = {label_id: label for label_id, label in
labels.items() if label_id != 'try_number'}
return ','.join([label_id + '=' + label for label_id, label in
sorted(filtered_labels.items())])
@staticmethod
-def _try_numbers_do_not_match(context, pod):
-return pod.metadata.labels['try_number'] != context['ti'].try_number
+def _try_numbers_match(context, pod):
+return pod.metadata.labels['try_number'] == context['ti'].try_number
@staticmethod
def _set_resources(resources):
[airflow] 01/32: Update some dependencies (#9684)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 0b5f0fc2c8717030a9de8555f3c8ccfa3875c9a0
Author: Hartorn
AuthorDate: Mon Jul 6 13:04:35 2020 +0200
Update some dependencies (#9684)
(cherry picked from commit fd62b1c5262086597db2aa439a09d86794a33345)
---
.github/workflows/ci.yml| 5 +-
breeze | 4 -
requirements/requirements-python2.7.txt | 132 +---
requirements/requirements-python3.5.txt | 26 +++
requirements/requirements-python3.6.txt | 54 +++--
requirements/requirements-python3.7.txt | 52 +++--
requirements/requirements-python3.8.txt | 51 ++--
requirements/setup-3.5.md5 | 2 +-
requirements/setup-3.6.md5 | 2 +-
requirements/setup-3.7.md5 | 2 +-
scripts/ci/ci_check_license.sh | 2 +-
scripts/ci/ci_fix_ownership.sh | 2 +-
scripts/ci/ci_flake8.sh | 2 +-
scripts/ci/ci_generate_requirements.sh | 2 +-
scripts/ci/ci_push_ci_image.sh | 2 +-
scripts/ci/ci_push_production_images.sh | 2 +-
scripts/ci/ci_run_static_checks.sh | 2 +-
setup.py| 6 +-
18 files changed, 180 insertions(+), 170 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index c37253f..029c341 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -32,7 +32,7 @@ env:
DB_RESET: "true"
VERBOSE: "true"
UPGRADE_TO_LATEST_REQUIREMENTS: "false"
- PYTHON_MAJOR_MINOR_VERSION: 3.5
+ PYTHON_MAJOR_MINOR_VERSION: 3.6
USE_GITHUB_REGISTRY: "true"
CACHE_IMAGE_PREFIX: ${{ github.repository }}
CACHE_REGISTRY_USERNAME: ${{ github.actor }}
@@ -76,7 +76,6 @@ jobs:
runs-on: ubuntu-latest
env:
CI_JOB_TYPE: "Documentation"
- PYTHON_MAJOR_MINOR_VERSION: 3.6
steps:
- uses: actions/checkout@master
- uses: actions/setup-python@v1
@@ -207,7 +206,7 @@ ${{
hashFiles('requirements/requirements-python${{matrix.python-version}}.txt')
- uses: actions/checkout@master
- uses: actions/setup-python@v1
with:
- python-version: '3.x'
+ python-version: '3.6'
- name: "Free space"
run: ./scripts/ci/ci_free_space_on_ci.sh
- name: "Build CI image ${{ matrix.python-version }}"
diff --git a/breeze b/breeze
index 3e8d2c3..534bec3 100755
--- a/breeze
+++ b/breeze
@@ -827,10 +827,6 @@ function parse_arguments() {
generate-requirements)
LAST_SUBCOMMAND="${1}"
COMMAND_TO_RUN="perform_generate_requirements"
- # if you want to generate requirement - you want to build the image
too :)
- export FORCE_ANSWER_TO_QUESTIONS="yes"
- # and assume you want to build it no matter if it is needed
- export FORCE_BUILD_IMAGES="true"
shift ;;
push-image)
LAST_SUBCOMMAND="${1}"
diff --git a/requirements/requirements-python2.7.txt
b/requirements/requirements-python2.7.txt
index 4f9242c..6973e5a 100644
--- a/requirements/requirements-python2.7.txt
+++ b/requirements/requirements-python2.7.txt
@@ -8,7 +8,7 @@ Flask-Caching==1.3.3
Flask-JWT-Extended==3.24.1
Flask-Login==0.4.1
Flask-OpenID==1.2.5
-Flask-SQLAlchemy==2.4.4
+Flask-SQLAlchemy==2.4.3
Flask-WTF==0.14.3
Flask==1.1.2
JPype1==0.7.1
@@ -23,9 +23,9 @@ PyNaCl==1.4.0
PySmbClient==0.1.5
PyYAML==5.3.1
Pygments==2.5.2
-SQLAlchemy-JSONField==0.9.0
+SQLAlchemy-JSONField==0.8.0
SQLAlchemy==1.3.18
-Sphinx==3.1.2
+Sphinx==1.8.5
Unidecode==1.1.1
WTForms==2.3.1
Werkzeug==0.16.1
@@ -36,37 +36,44 @@ amqp==2.6.0
analytics-python==1.2.9
ansiwrap==0.8.4
apipkg==1.5
-apispec==3.3.1
+apispec==2.0.2
appdirs==1.4.4
-argcomplete==1.12.0
+argcomplete==1.11.1
asn1crypto==1.3.0
aspy.yaml==1.3.0
-astroid==2.4.2
+astroid==1.6.6
atlasclient==1.0.0
atomicwrites==1.4.0
attrs==19.3.0
aws-sam-translator==1.25.0
aws-xray-sdk==2.6.0
azure-common==1.1.25
-azure-cosmos==3.2.0
+azure-cosmos==3.1.2
azure-datalake-store==0.0.48
azure-mgmt-containerinstance==1.5.0
-azure-mgmt-resource==10.1.0
+azure-mgmt-nspkg==3.0.2
+azure-mgmt-resource==10.0.0
azure-nspkg==3.0.2
azure-storage-blob==2.1.0
azure-storage-common==2.1.0
+azure-storage-nspkg==3.1.0
azure-storage==0.36.0
-backcall==0.2.0
+backports-abc==0.5
+backports.functools-lru-cache==1.6.1
+backports.shutil-get-terminal-size==1.0.0
+backports.ssl-match-hostname==3.7.0.1
+backports.tempfile==1.0
+backports.weakref==1.0.post1
bcrypt==3.1.7
beautifulsoup4==4.7.1
billiard==3.6.3.0
bleach==3.1.5
blinker==1.4
-boto3==1.14.20
+boto3==1.14.14
boto==2.49.0
-botocore==1.17.20
+botocore==1.17.14
cached-property==1.5.1
-cachetools==4.1.1
+cachetools==3.1.1
cassandra-driver==3.20.2
cattrs==1.0.0
celery==4.4.6
@@ -81,11 +88,14 @@ cloudant==0.5.10
[airflow] 06/32: The group of embedded DAGs should be root to be OpenShift compatible (#9794)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit d61c33db6118c1f0e16341e2dd8b20557c041677
Author: Jarek Potiuk
AuthorDate: Mon Jul 13 20:47:55 2020 +0200
The group of embedded DAGs should be root to be OpenShift compatible (#9794)
(cherry picked from commit 8f6b8378aa46c8226b8dd56c509affe7f2b5a4bc)
---
Dockerfile | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Dockerfile b/Dockerfile
index 89225b8..a882178 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -358,7 +358,7 @@ COPY scripts/prod/clean-logs.sh /clean-logs
ARG EMBEDDED_DAGS="empty"
-COPY --chown=airflow:airflow ${EMBEDDED_DAGS}/ ${AIRFLOW_HOME}/dags/
+COPY --chown=airflow:root ${EMBEDDED_DAGS}/ ${AIRFLOW_HOME}/dags/
RUN chmod a+x /entrypoint /clean-logs
[airflow] 04/32: [AIRFLOW-5391] Do not re-run skipped tasks when they are cleared (#7276)
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 179e93093b7b21fa60cf8d5984ec4fd0c6b52730 Author: yuqian90 AuthorDate: Fri Feb 21 19:35:55 2020 +0800 [AIRFLOW-5391] Do not re-run skipped tasks when they are cleared (#7276) If a task is skipped by BranchPythonOperator, BaseBranchOperator or ShortCircuitOperator and the user then clears the skipped task later, it'll execute. This is probably not the right behaviour. This commit changes that so it will be skipped again. This can be ignored by running the task again with "Ignore Task Deps" override. (cherry picked from commit 1cdab56a6192f69962506b7ff632c986c84eb10d) --- UPDATING.md| 7 ++ airflow/models/baseoperator.py | 2 + airflow/models/skipmixin.py| 112 - airflow/ti_deps/dep_context.py | 27 - airflow/ti_deps/deps/not_previously_skipped_dep.py | 88 ++ requirements/requirements-python2.7.txt| 67 ++- requirements/requirements-python3.8.txt| 92 +++--- requirements/setup-2.7.md5 | 2 +- requirements/setup-3.8.md5 | 2 +- tests/jobs/test_scheduler_job.py | 39 ++ tests/operators/test_latest_only_operator.py | 60 ++ tests/operators/test_python_operator.py| 119 +- .../deps/test_not_previously_skipped_dep.py| 133 + tests/ti_deps/deps/test_trigger_rule_dep.py| 40 +++ 14 files changed, 655 insertions(+), 135 deletions(-) diff --git a/UPDATING.md b/UPDATING.md index 61734bb..f82ba10 100644 --- a/UPDATING.md +++ b/UPDATING.md @@ -25,6 +25,7 @@ assists users migrating to a new version. **Table of contents** +- [Airflow 1.10.12](#airflow-11012) - [Airflow 1.10.11](#airflow-11011) - [Airflow 1.10.10](#airflow-11010) - [Airflow 1.10.9](#airflow-1109) @@ -59,6 +60,12 @@ More tips can be found in the guide: https://developers.google.com/style/inclusive-documentation --> +## Airflow 1.10.12 + +### Clearing tasks skipped by SkipMixin will skip them + +Previously, when tasks skipped by SkipMixin (such as BranchPythonOperator, BaseBranchOperator and ShortCircuitOperator) are cleared, they execute. Since 1.10.12, when such skipped tasks are cleared, +they will be skipped again by the newly introduced NotPreviouslySkippedDep. ## Airflow 1.10.11 diff --git a/airflow/models/baseoperator.py b/airflow/models/baseoperator.py index 52037c5..266ad64 100644 --- a/airflow/models/baseoperator.py +++ b/airflow/models/baseoperator.py @@ -45,6 +45,7 @@ from airflow.models.pool import Pool from airflow.models.taskinstance import TaskInstance, clear_task_instances from airflow.models.xcom import XCOM_RETURN_KEY from airflow.ti_deps.deps.not_in_retry_period_dep import NotInRetryPeriodDep +from airflow.ti_deps.deps.not_previously_skipped_dep import NotPreviouslySkippedDep from airflow.ti_deps.deps.prev_dagrun_dep import PrevDagrunDep from airflow.ti_deps.deps.trigger_rule_dep import TriggerRuleDep from airflow.utils import timezone @@ -575,6 +576,7 @@ class BaseOperator(LoggingMixin): NotInRetryPeriodDep(), PrevDagrunDep(), TriggerRuleDep(), +NotPreviouslySkippedDep(), } @property diff --git a/airflow/models/skipmixin.py b/airflow/models/skipmixin.py index 57341d8..3b4531f 100644 --- a/airflow/models/skipmixin.py +++ b/airflow/models/skipmixin.py @@ -19,28 +19,28 @@ from airflow.models.taskinstance import TaskInstance from airflow.utils import timezone -from airflow.utils.db import provide_session +from airflow.utils.db import create_session, provide_session from airflow.utils.log.logging_mixin import LoggingMixin from airflow.utils.state import State import six -from typing import Union, Iterable, Set +from typing import Set + +# The key used by SkipMixin to store XCom data. +XCOM_SKIPMIXIN_KEY = "skipmixin_key" + +# The dictionary key used to denote task IDs that are skipped +XCOM_SKIPMIXIN_SKIPPED = "skipped" + +# The dictionary key used to denote task IDs that are followed +XCOM_SKIPMIXIN_FOLLOWED = "followed" class SkipMixin(LoggingMixin): -@provide_session -def skip(self, dag_run, execution_date, tasks, session=None): +def _set_state_to_skipped(self, dag_run, execution_date, tasks, session): """ -Sets tasks instances to skipped from the same dag run. - -:param dag_run: the DagRun for which to set the tasks to skipped -:param execution_date: execution_date -:param tasks: tasks to skip (not task_ids) -:param session: db session to use +Used internally to set state of task instances to skipped from
[airflow] 24/32: Pin Pyarrow < 1.0
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 70a741601646dd68b5e951f7a46cc591fda52020 Author: Kaxil Naik AuthorDate: Sun Aug 2 12:19:42 2020 +0100 Pin Pyarrow < 1.0 --- setup.py | 1 + 1 file changed, 1 insertion(+) diff --git a/setup.py b/setup.py index cc0f721..327e157 100644 --- a/setup.py +++ b/setup.py @@ -325,6 +325,7 @@ pagerduty = [ papermill = [ 'papermill[all]>=1.0.0', 'nteract-scrapbook[all]>=0.2.1', +'pyarrow<1.0.0' ] password = [ 'bcrypt>=2.0.0',
[airflow] 32/32: Makes multi-namespace mode optional (#9570)
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 242d6d0e9a1955b11677d2be1b7ae5e28243e619 Author: Daniel Imberman AuthorDate: Mon Aug 10 13:41:40 2020 -0700 Makes multi-namespace mode optional (#9570) Running the airflow k8sexecutor with multiple namespace abilities requires creating a ClusterRole which can break existing deployments Co-authored-by: Daniel Imberman (cherry picked from commit 2e3c878066f9241d17f2e4ba41fe0e2ba02de79e) --- airflow/config_templates/config.yml | 7 ++ airflow/config_templates/default_airflow.cfg | 4 airflow/executors/kubernetes_executor.py | 32 +++- 3 files changed, 38 insertions(+), 5 deletions(-) diff --git a/airflow/config_templates/config.yml b/airflow/config_templates/config.yml index f54255e..75c47cb 100644 --- a/airflow/config_templates/config.yml +++ b/airflow/config_templates/config.yml @@ -1812,6 +1812,13 @@ type: string example: ~ default: "default" +- name: multi_namespace_mode + description: | +Allows users to launch pods in multiple namespaces. +Will require creating a cluster-role for the scheduler + type: boolean + example: ~ + default: "False" - name: airflow_configmap description: | The name of the Kubernetes ConfigMap containing the Airflow Configuration (this file) diff --git a/airflow/config_templates/default_airflow.cfg b/airflow/config_templates/default_airflow.cfg index e18e538..3a9bba2 100644 --- a/airflow/config_templates/default_airflow.cfg +++ b/airflow/config_templates/default_airflow.cfg @@ -838,6 +838,10 @@ worker_pods_creation_batch_size = 1 # The Kubernetes namespace where airflow workers should be created. Defaults to ``default`` namespace = default +# Allows users to launch pods in multiple namespaces. +# Will require creating a cluster-role for the scheduler +multi_namespace_mode = False + # The name of the Kubernetes ConfigMap containing the Airflow Configuration (this file) # Example: airflow_configmap = airflow-configmap airflow_configmap = diff --git a/airflow/executors/kubernetes_executor.py b/airflow/executors/kubernetes_executor.py index 3ad4222..7b31b45 100644 --- a/airflow/executors/kubernetes_executor.py +++ b/airflow/executors/kubernetes_executor.py @@ -22,6 +22,7 @@ KubernetesExecutor :ref:`executor:KubernetesExecutor` """ import base64 +import functools import json import multiprocessing import time @@ -162,6 +163,7 @@ class KubeConfig: # cluster has RBAC enabled, your scheduler may need service account permissions to # create, watch, get, and delete pods in this namespace. self.kube_namespace = conf.get(self.kubernetes_section, 'namespace') +self.multi_namespace_mode = conf.get(self.kubernetes_section, 'multi_namespace_mode') # The Kubernetes Namespace in which pods will be created by the executor. Note # that if your # cluster has RBAC enabled, your workers may need service account permissions to @@ -254,9 +256,17 @@ class KubeConfig: class KubernetesJobWatcher(multiprocessing.Process, LoggingMixin): """Watches for Kubernetes jobs""" -def __init__(self, namespace, watcher_queue, resource_version, worker_uuid, kube_config): + +def __init__(self, + namespace, + mult_namespace_mode, + watcher_queue, + resource_version, + worker_uuid, + kube_config): multiprocessing.Process.__init__(self) self.namespace = namespace +self.multi_namespace_mode = mult_namespace_mode self.worker_uuid = worker_uuid self.watcher_queue = watcher_queue self.resource_version = resource_version @@ -295,8 +305,16 @@ class KubernetesJobWatcher(multiprocessing.Process, LoggingMixin): kwargs[key] = value last_resource_version = None -for event in watcher.stream(kube_client.list_namespaced_pod, self.namespace, -**kwargs): +if self.multi_namespace_mode: +list_worker_pods = functools.partial(watcher.stream, + kube_client.list_pod_for_all_namespaces, + **kwargs) +else: +list_worker_pods = functools.partial(watcher.stream, + kube_client.list_namespaced_pod, + self.namespace, + **kwargs) +for event in list_worker_pods(): task = event['object'] self.log.info( 'Event: %s had an event of type %s', @@ -377,8 +395,12 @@
[airflow] 12/32: Constraint files are now maintained automatically (#9889)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 5f93baf3f8a785b93b6ee9811d3938d8200c55ad
Author: Jarek Potiuk
AuthorDate: Mon Jul 20 14:36:03 2020 +0200
Constraint files are now maintained automatically (#9889)
* Constraint files are now maintained automatically
* No need to generate requirements when setup.py changes
* requirements are kept in separate orphan branches not in main repo
* merges to master verify if latest requirements are working and
push tested requirements to orphaned branches
* we keep history of requirement changes and can label them
individually for each version (by constraint-1.10.n tag name)
* consistently changed all references to be 'constraints' not
'requirements'
(cherry picked from commit de9eaeb434747897a192ef31815fbdd519e29c4d)
---
.dockerignore | 1 -
.github/workflows/ci.yml | 147 +
BREEZE.rst | 64 +
CI.rst | 113 +---
CONTRIBUTING.rst | 71 --
Dockerfile | 17 +--
Dockerfile.ci | 21 +--
IMAGES.rst | 75 +--
INSTALL| 7 +-
LOCAL_VIRTUALENV.rst | 13 +-
README.md | 17 +--
breeze | 41 +++---
breeze-complete| 2 +-
common/_default_branch.sh | 1 +
docs/installation.rst | 31 +++--
requirements/REMOVE.md | 22 +++
.../ci_generate_constraints.sh}| 2 +-
scripts/ci/docker-compose/local.yml| 1 -
.../ci/in_container/run_generate_constraints.sh| 50 +++
.../ci/in_container/run_generate_requirements.sh | 80 ---
scripts/ci/kubernetes/ci_run_kubernetes_tests.sh | 5 +-
scripts/ci/libraries/_build_images.sh | 34 ++---
scripts/ci/libraries/_initialization.sh| 16 +--
scripts/ci/libraries/_local_mounts.sh | 1 -
scripts/ci/libraries/_runs.sh | 8 +-
.../pre_commit/pre_commit_generate_requirements.sh | 24
scripts/ci/static_checks/ci_run_static_checks.sh | 3 +
.../ci/tools/ci_check_if_tests_should_be_run.sh| 1 -
28 files changed, 444 insertions(+), 424 deletions(-)
diff --git a/.dockerignore b/.dockerignore
index 6f89516..d7d621d 100644
--- a/.dockerignore
+++ b/.dockerignore
@@ -46,7 +46,6 @@
!MANIFEST.in
!NOTICE
!.github
-!requirements
!empty
# Avoid triggering context change on README change (new companies using
Airflow)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 604fa0d..aac8be1 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -31,7 +31,7 @@ env:
SKIP_CI_IMAGE_CHECK: "true"
DB_RESET: "true"
VERBOSE: "true"
- UPGRADE_TO_LATEST_REQUIREMENTS: "false"
+ UPGRADE_TO_LATEST_CONSTRAINTS: ${{ github.event_name == 'push' ||
github.event_name == 'scheduled' }}
PYTHON_MAJOR_MINOR_VERSION: 3.6
USE_GITHUB_REGISTRY: "true"
CACHE_IMAGE_PREFIX: ${{ github.repository }}
@@ -66,7 +66,6 @@ jobs:
- cancel-previous-workflow-run
env:
MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
- CI_JOB_TYPE: "Static checks"
steps:
- uses: actions/checkout@master
- uses: actions/setup-python@v1
@@ -84,19 +83,13 @@ jobs:
- name: "Build CI image"
run: ./scripts/ci/images/ci_prepare_ci_image_on_ci.sh
- name: "Static checks"
-if: success()
-run: |
- python -m pip install pre-commit \
- --constraint
requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt
- ./scripts/ci/static_checks/ci_run_static_checks.sh
+run: ./scripts/ci/static_checks/ci_run_static_checks.sh
docs:
timeout-minutes: 60
name: "Build docs"
runs-on: ubuntu-latest
needs:
- cancel-previous-workflow-run
-env:
- CI_JOB_TYPE: "Documentation"
steps:
- uses: actions/checkout@master
- uses: actions/setup-python@v1
@@ -142,7 +135,6 @@ jobs:
BACKEND: postgres
TEST_TYPE: ${{ matrix.test-type }}
RUN_TESTS: "true"
- CI_JOB_TYPE: "Tests"
SKIP_CI_IMAGE_CHECK: "true"
RUNTIME: "kubernetes"
ENABLE_KIND_CLUSTER: "true"
@@ -173,8 +165,7 @@ jobs:
cache-name: cache-kubernetes-tests-virtualenv-v4
with:
path: .build/.kubernetes_venv
- key: "${{
[airflow] 09/32: Group CI scripts in subdirectories (#9653)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 7ec2b3ace1a1a4982b0c313b89eab6aee1eb9620
Author: Jarek Potiuk
AuthorDate: Thu Jul 16 18:05:35 2020 +0200
Group CI scripts in subdirectories (#9653)
Reviewed the scripts and removed some of the old unused ones.
(cherry picked from commit faec41ec9a05a037b88fd0213b1936cde2b5c454)
---
.github/workflows/ci.yml | 62 +-
.pre-commit-config.yaml| 34 +++---
.rat-excludes | 1 +
BREEZE.rst | 2 +-
STATIC_CODE_CHECKS.rst | 15 ++-
TESTING.rst| 8 +-
breeze | 4 +-
docs/start_doc_server.sh | 4 +-
hooks/build| 4 +-
hooks/push | 5 -
scripts/ci/ci_load_image_to_kind.sh| 33 -
scripts/ci/ci_perform_kind_cluster_operation.sh| 32 -
scripts/ci/{ => docs}/ci_docs.sh | 4 +-
scripts/ci/{ => images}/ci_build_dockerhub.sh | 4 +-
.../ci/{ => images}/ci_prepare_ci_image_on_ci.sh | 4 +-
.../ci/{ => images}/ci_prepare_prod_image_on_ci.sh | 4 +-
scripts/ci/{ => images}/ci_push_ci_image.sh| 4 +-
.../ci/{ => images}/ci_push_production_images.sh | 4 +-
.../ci/in_container/_in_container_script_init.sh | 4 +-
scripts/ci/in_container/_in_container_utils.sh | 2 +-
.../in_container/deploy_airflow_to_kubernetes.sh | 23
scripts/ci/in_container/entrypoint_ci.sh | 16 +--
scripts/ci/{ => in_container}/run_cli_tool.sh | 0
scripts/ci/in_container/run_system_tests.sh| 4 +-
.../ci_deploy_app_to_kubernetes.sh | 4 +-
.../ci/{ => kubernetes}/ci_run_kubernetes_tests.sh | 4 +-
scripts/ci/{ => libraries}/_all_libs.sh| 29 +++--
scripts/ci/libraries/_build_images.sh | 2 +-
scripts/ci/libraries/_initialization.sh| 6 +-
scripts/ci/libraries/_kind.sh | 40 +++---
scripts/ci/{ => libraries}/_script_init.sh | 13 +-
scripts/ci/minikdc.properties | 27 -
.../ci/{ => pre_commit}/pre_commit_bat_tests.sh| 4 +-
.../{ => pre_commit}/pre_commit_breeze_cmd_line.sh | 10 +-
.../pre_commit_check_integrations.sh | 10 +-
.../{ => pre_commit}/pre_commit_check_license.sh | 2 +-
.../ci/pre_commit/pre_commit_check_order_setup.py | 135 +
scripts/ci/{ => pre_commit}/pre_commit_ci_build.sh | 4 +-
scripts/ci/{ => pre_commit}/pre_commit_flake8.sh | 4 +-
.../pre_commit_generate_requirements.sh| 4 +-
.../ci/pre_commit/pre_commit_insert_extras.py | 4 +-
.../{ => pre_commit}/pre_commit_lint_dockerfile.sh | 4 +-
.../pre_commit_local_yml_mounts.sh | 8 +-
scripts/ci/{ => pre_commit}/pre_commit_mypy.sh | 4 +-
.../ci/{ => pre_commit}/pre_commit_yaml_to_cfg.py | 7 +-
scripts/ci/pre_commit_update_extras.sh | 31 -
.../{ => requirements}/ci_generate_requirements.sh | 4 +-
scripts/ci/{ => static_checks}/ci_bat_tests.sh | 3 +
scripts/ci/{ => static_checks}/ci_check_license.sh | 4 +-
scripts/ci/{ => static_checks}/ci_flake8.sh| 4 +-
.../ci/{ => static_checks}/ci_lint_dockerfile.sh | 4 +-
scripts/ci/{ => static_checks}/ci_mypy.sh | 4 +-
.../ci/{ => static_checks}/ci_run_static_checks.sh | 4 +-
scripts/ci/{ => testing}/ci_run_airflow_testing.sh | 33 ++---
scripts/ci/{ => tools}/ci_count_changed_files.sh | 4 +-
scripts/ci/{ => tools}/ci_fix_ownership.sh | 10 +-
scripts/ci/{ => tools}/ci_free_space_on_ci.sh | 4 +-
tests/bats/bats_utils.bash | 4 +-
tests/test_order_setup.py | 134
59 files changed, 355 insertions(+), 494 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 134bc1f..a849d58 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -60,15 +60,15 @@ jobs:
path: ~/.cache/pre-commit
key: ${{ env.cache-name }}-${{ github.job }}-${{
hashFiles('.pre-commit-config.yaml') }}
- name: "Free space"
-run: ./scripts/ci/ci_free_space_on_ci.sh
+run: ./scripts/ci/tools/ci_free_space_on_ci.sh
- name: "Build CI image"
-run: ./scripts/ci/ci_prepare_ci_image_on_ci.sh
+run: ./scripts/ci/images/ci_prepare_ci_image_on_ci.sh
- name: "Static checks"
if: success()
run: |
python -m pip install pre-commit \
--constraint
[airflow] 28/32: Add getimport for xcom change
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 1a8ba6a95696a19eb422a3d0a9553b3a59827052
Author: Daniel Imberman
AuthorDate: Mon Aug 3 15:59:40 2020 -0700
Add getimport for xcom change
---
airflow/configuration.py | 21 +
1 file changed, 21 insertions(+)
diff --git a/airflow/configuration.py b/airflow/configuration.py
index d912898..01ee90f 100644
--- a/airflow/configuration.py
+++ b/airflow/configuration.py
@@ -42,6 +42,7 @@ import yaml
from zope.deprecation import deprecated
from airflow.exceptions import AirflowConfigException
+from airflow.utils.module_loading import import_string
standard_library.install_aliases()
@@ -342,6 +343,26 @@ class AirflowConfigParser(ConfigParser):
"section/key [{section}/{key}] not found "
"in config".format(section=section, key=key))
+def getimport(self, section, key, **kwargs):
+"""
+Reads options, imports the full qualified name, and returns the object.
+In case of failure, it throws an exception a clear message with the
key aad the section names
+:return: The object or None, if the option is empty
+"""
+full_qualified_path = conf.get(section=section, key=key, **kwargs)
+if not full_qualified_path:
+return None
+
+try:
+return import_string(full_qualified_path)
+except ImportError as e:
+log.error(e)
+raise AirflowConfigException(
+'The object could not be loaded. Please check "{key}" key in
"{section}" section. '
+'Current value: "{full_qualified_path}".' .format(
+key=key, section=section,
full_qualified_path=full_qualified_path)
+)
+
def getboolean(self, section, key, **kwargs):
val = str(self.get(section, key, **kwargs)).lower().strip()
if '#' in val:
[airflow] 25/32: Fix docstrings in BigQueryGetDataOperator (#10042)
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 06b06d77e17ec29d74c59f0dae8da33698e286bb Author: Jinhui Zhang AuthorDate: Wed Jul 29 05:42:41 2020 -0700 Fix docstrings in BigQueryGetDataOperator (#10042) (cherry picked from commit 59cbff0874dd5318cda4b9ce7b7eeb1aad1dad4d) --- airflow/contrib/operators/bigquery_get_data.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/airflow/contrib/operators/bigquery_get_data.py b/airflow/contrib/operators/bigquery_get_data.py index f5e6e50..e16804b 100644 --- a/airflow/contrib/operators/bigquery_get_data.py +++ b/airflow/contrib/operators/bigquery_get_data.py @@ -56,7 +56,7 @@ class BigQueryGetDataOperator(BaseOperator): :type table_id: str :param max_results: The maximum number of records (rows) to be fetched from the table. (templated) -:type max_results: str +:type max_results: int :param selected_fields: List of fields to return (comma-separated). If unspecified, all fields are returned. :type selected_fields: str @@ -74,7 +74,7 @@ class BigQueryGetDataOperator(BaseOperator): def __init__(self, dataset_id, table_id, - max_results='100', + max_results=100, selected_fields=None, bigquery_conn_id='bigquery_default', delegate_to=None,
[airflow] 08/32: For now cloud tools are not needed in CI (#9818)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 1a418797eb650cae501ca5460a142b055e335842
Author: Jarek Potiuk
AuthorDate: Tue Jul 14 16:35:33 2020 +0200
For now cloud tools are not needed in CI (#9818)
Currently there is "unbound" variable error printed in CI logs
because of that.
(cherry picked from commit 69f82e66af54fb85a07ee6c7c85b8d4f5140e758)
---
scripts/ci/in_container/entrypoint_ci.sh | 8 +---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/scripts/ci/in_container/entrypoint_ci.sh
b/scripts/ci/in_container/entrypoint_ci.sh
index 349b092..4d1bf0c 100755
--- a/scripts/ci/in_container/entrypoint_ci.sh
+++ b/scripts/ci/in_container/entrypoint_ci.sh
@@ -45,9 +45,11 @@ RUN_TESTS=${RUN_TESTS:="false"}
CI=${CI:="false"}
INSTALL_AIRFLOW_VERSION="${INSTALL_AIRFLOW_VERSION:=""}"
-# Create links for useful CLI tools
-# shellcheck source=scripts/ci/run_cli_tool.sh
-source <(bash scripts/ci/run_cli_tool.sh)
+if [[ ${CI} == "false" ]]; then
+# Create links for useful CLI tools
+# shellcheck source=scripts/ci/run_cli_tool.sh
+source <(bash scripts/ci/run_cli_tool.sh)
+fi
if [[ ${AIRFLOW_VERSION} == *1.10* || ${INSTALL_AIRFLOW_VERSION} == *1.10* ]];
then
export RUN_AIRFLOW_1_10="true"
[airflow] 21/32: Add pre 1.10.11 Kubernetes Paths back with Deprecation Warning (#10067)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit bfa089d4adff5a0892c7e2ef2c9639938e86cdb1
Author: Kaxil Naik
AuthorDate: Fri Jul 31 09:34:18 2020 +0100
Add pre 1.10.11 Kubernetes Paths back with Deprecation Warning (#10067)
---
airflow/contrib/kubernetes/__init__.py | 2 --
.../contrib/kubernetes/{__init__.py => kube_client.py} | 14 ++
.../kubernetes}/pod.py | 17 +++--
.../kubernetes/{__init__.py => pod_runtime_info_env.py} | 14 ++
.../kubernetes/{__init__.py => refresh_config.py} | 16
.../__init__.py => contrib/kubernetes/secret.py}| 11 +++
airflow/contrib/kubernetes/{__init__.py => volume.py} | 14 ++
.../contrib/kubernetes/{__init__.py => volume_mount.py} | 14 ++
airflow/kubernetes/pod_launcher.py | 5 +
airflow/kubernetes/pod_launcher_helper.py | 2 +-
airflow/kubernetes/pod_runtime_info_env.py | 2 +-
airflow/kubernetes/secret.py| 4 +++-
airflow/kubernetes/volume_mount.py | 3 +--
tests/kubernetes/models/test_pod.py | 2 +-
tests/kubernetes/test_pod_launcher_helper.py| 5 +++--
15 files changed, 93 insertions(+), 32 deletions(-)
diff --git a/airflow/contrib/kubernetes/__init__.py
b/airflow/contrib/kubernetes/__init__.py
index ef7074c..b7f8352 100644
--- a/airflow/contrib/kubernetes/__init__.py
+++ b/airflow/contrib/kubernetes/__init__.py
@@ -17,5 +17,3 @@
# specific language governing permissions and limitations
# under the License.
#
-
-from airflow.kubernetes import * # noqa
diff --git a/airflow/contrib/kubernetes/__init__.py
b/airflow/contrib/kubernetes/kube_client.py
similarity index 71%
copy from airflow/contrib/kubernetes/__init__.py
copy to airflow/contrib/kubernetes/kube_client.py
index ef7074c..d785fac 100644
--- a/airflow/contrib/kubernetes/__init__.py
+++ b/airflow/contrib/kubernetes/kube_client.py
@@ -1,5 +1,3 @@
-# -*- coding: utf-8 -*-
-#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -16,6 +14,14 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-#
+"""This module is deprecated. Please use `airflow.kubernetes.kube_client`."""
+
+import warnings
+
+# pylint: disable=unused-import
+from airflow.kubernetes.kube_client import * # noqa
-from airflow.kubernetes import * # noqa
+warnings.warn(
+"This module is deprecated. Please use `airflow.kubernetes.kube_client`.",
+DeprecationWarning, stacklevel=2
+)
diff --git a/airflow/kubernetes_deprecated/pod.py
b/airflow/contrib/kubernetes/pod.py
similarity index 92%
rename from airflow/kubernetes_deprecated/pod.py
rename to airflow/contrib/kubernetes/pod.py
index 22a8c12..0ab3616 100644
--- a/airflow/kubernetes_deprecated/pod.py
+++ b/airflow/contrib/kubernetes/pod.py
@@ -14,9 +14,17 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+"""This module is deprecated. Please use `airflow.kubernetes.pod`."""
-import kubernetes.client.models as k8s
-from airflow.kubernetes.pod import Resources
+import warnings
+
+# pylint: disable=unused-import
+from airflow.kubernetes.pod import Port, Resources # noqa
+
+warnings.warn(
+"This module is deprecated. Please use `airflow.kubernetes.pod`.",
+DeprecationWarning, stacklevel=2
+)
class Pod(object):
@@ -86,6 +94,10 @@ class Pod(object):
pod_runtime_info_envs=None,
dnspolicy=None
):
+warnings.warn(
+"Using `airflow.contrib.kubernetes.pod.Pod` is deprecated. Please
use `k8s.V1Pod`.",
+DeprecationWarning, stacklevel=2
+)
self.image = image
self.envs = envs or {}
self.cmds = cmds
@@ -119,6 +131,7 @@ class Pod(object):
:return: k8s.V1Pod
"""
+import kubernetes.client.models as k8s
meta = k8s.V1ObjectMeta(
labels=self.labels,
name=self.name,
diff --git a/airflow/contrib/kubernetes/__init__.py
b/airflow/contrib/kubernetes/pod_runtime_info_env.py
similarity index 68%
copy from airflow/contrib/kubernetes/__init__.py
copy to airflow/contrib/kubernetes/pod_runtime_info_env.py
index ef7074c..0dc8aed 100644
--- a/airflow/contrib/kubernetes/__init__.py
+++ b/airflow/contrib/kubernetes/pod_runtime_info_env.py
@@ -1,5 +1,3 @@
-# -*- coding: utf-8 -*-
-#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE
[airflow] 30/32: Fix more PodMutationHook issues for backwards compatibility (#10084)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit c230156739178762d5cef482ace3d7a05e683cc1
Author: Kaxil Naik
AuthorDate: Fri Aug 7 11:50:44 2020 +0100
Fix more PodMutationHook issues for backwards compatibility (#10084)
Co-authored-by: Daniel Imberman
---
UPDATING.md | 10 +
airflow/contrib/executors/kubernetes_executor.py | 20 +
airflow/contrib/kubernetes/pod.py| 143 ++-
airflow/executors/kubernetes_executor.py | 6 +
airflow/kubernetes/pod.py| 31 +-
airflow/kubernetes/pod_generator.py | 76 +++-
airflow/kubernetes/pod_launcher.py | 73 +++-
airflow/kubernetes/pod_launcher_helper.py| 96 -
airflow/kubernetes/secret.py | 21 +-
airflow/kubernetes/volume.py | 17 +-
airflow/operators/python_operator.py | 4 +-
docs/conf.py | 1 +
kubernetes_tests/test_kubernetes_pod_operator.py | 1 -
tests/kubernetes/models/test_pod.py | 108 +++---
tests/kubernetes/models/test_volume.py | 40 ++
tests/kubernetes/test_pod_generator.py | 206 +-
tests/kubernetes/test_pod_launcher.py| 153 +++-
tests/kubernetes/test_pod_launcher_helper.py | 98 -
tests/kubernetes/test_worker_configuration.py| 7 +
tests/test_local_settings.py | 269 -
tests/test_local_settings/__init__.py| 16 +
tests/test_local_settings/test_local_settings.py | 461 +++
22 files changed, 1289 insertions(+), 568 deletions(-)
diff --git a/UPDATING.md b/UPDATING.md
index f82ba10..4f2b844 100644
--- a/UPDATING.md
+++ b/UPDATING.md
@@ -67,6 +67,16 @@ https://developers.google.com/style/inclusive-documentation
Previously, when tasks skipped by SkipMixin (such as BranchPythonOperator,
BaseBranchOperator and ShortCircuitOperator) are cleared, they execute. Since
1.10.12, when such skipped tasks are cleared,
they will be skipped again by the newly introduced NotPreviouslySkippedDep.
+### The pod_mutation_hook function will now accept a kubernetes V1Pod object
+
+As of airflow 1.10.12, using the `airflow.contrib.kubernetes.Pod` class in the
`pod_mutation_hook` is now deprecated. Instead we recommend that users
+treat the `pod` parameter as a `kubernetes.client.models.V1Pod` object. This
means that users now have access to the full Kubernetes API
+when modifying airflow pods
+
+### pod_template_file option now available in the KubernetesPodOperator
+
+Users can now offer a path to a yaml for the KubernetesPodOperator using the
`pod_template_file` parameter.
+
## Airflow 1.10.11
### Use NULL as default value for dag.description
diff --git a/airflow/contrib/executors/kubernetes_executor.py
b/airflow/contrib/executors/kubernetes_executor.py
new file mode 100644
index 000..416b2d7
--- /dev/null
+++ b/airflow/contrib/executors/kubernetes_executor.py
@@ -0,0 +1,20 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.executors import kubernetes_executor # noqa
diff --git a/airflow/contrib/kubernetes/pod.py
b/airflow/contrib/kubernetes/pod.py
index 0ab3616..944cd8c 100644
--- a/airflow/contrib/kubernetes/pod.py
+++ b/airflow/contrib/kubernetes/pod.py
@@ -19,7 +19,18 @@
import warnings
# pylint: disable=unused-import
-from airflow.kubernetes.pod import Port, Resources # noqa
+from typing import List, Union
+
+from kubernetes.client import models as k8s
+
+from airflow.kubernetes.pod import Port, Resources # noqa
+from airflow.kubernetes.volume import Volume
+from airflow.kubernetes.volume_mount import VolumeMount
+from airflow.kubernetes.secret import Secret
+
+from kubernetes.client.api_client import ApiClient
+
+api_client = ApiClient()
warnings.warn(
"This module is deprecated. Please use `airflow.kubernetes.pod`.",
@@ -120,7 +131,7 @@ class Pod(object):
self.affinity = affinity or {}
self.hostnetwork = hostnetwork or False
[airflow] 20/32: Fixes PodMutationHook for backwards compatibility (#9903)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit bcd02ddb81a07026dcbbc5e5a4dc669a6483b59b
Author: Daniel Imberman
AuthorDate: Thu Jul 30 11:40:23 2020 -0700
Fixes PodMutationHook for backwards compatibility (#9903)
Co-authored-by: Daniel Imberman
Co-authored-by: Kaxil Naik
---
airflow/kubernetes/k8s_model.py | 16 +++
airflow/kubernetes/pod.py| 33 --
airflow/kubernetes/pod_launcher.py | 26 +++-
airflow/kubernetes/pod_launcher_helper.py| 96 +++
airflow/kubernetes/volume_mount.py | 1 +
airflow/kubernetes_deprecated/__init__.py| 16 +++
airflow/kubernetes_deprecated/pod.py | 171 +++
docs/conf.py | 1 +
tests/kubernetes/models/test_pod.py | 81 +
tests/kubernetes/test_pod_launcher_helper.py | 97 +++
tests/test_local_settings.py | 96 +++
11 files changed, 619 insertions(+), 15 deletions(-)
diff --git a/airflow/kubernetes/k8s_model.py b/airflow/kubernetes/k8s_model.py
index 3fd2f9e..e10a946 100644
--- a/airflow/kubernetes/k8s_model.py
+++ b/airflow/kubernetes/k8s_model.py
@@ -29,6 +29,7 @@ else:
class K8SModel(ABC):
+
"""
These Airflow Kubernetes models are here for backwards compatibility
reasons only. Ideally clients should use the kubernetes api
@@ -39,6 +40,7 @@ class K8SModel(ABC):
can be avoided. All of these models implement the
`attach_to_pod` method so that they integrate with the kubernetes client.
"""
+
@abc.abstractmethod
def attach_to_pod(self, pod):
"""
@@ -47,9 +49,23 @@ class K8SModel(ABC):
:return: The pod with the object attached
"""
+def as_dict(self):
+res = {}
+if hasattr(self, "__slots__"):
+for s in self.__slots__:
+if hasattr(self, s):
+res[s] = getattr(self, s)
+if hasattr(self, "__dict__"):
+res_dict = self.__dict__.copy()
+res_dict.update(res)
+return res_dict
+return res
+
def append_to_pod(pod, k8s_objects):
"""
+Attach Kubernetes objects to the given POD
+
:param pod: A pod to attach a list of Kubernetes objects to
:type pod: kubernetes.client.models.V1Pod
:param k8s_objects: a potential None list of K8SModels
diff --git a/airflow/kubernetes/pod.py b/airflow/kubernetes/pod.py
index 0b332c2..9e455af 100644
--- a/airflow/kubernetes/pod.py
+++ b/airflow/kubernetes/pod.py
@@ -26,7 +26,13 @@ from airflow.kubernetes.k8s_model import K8SModel
class Resources(K8SModel):
-__slots__ = ('request_memory', 'request_cpu', 'limit_memory', 'limit_cpu',
'limit_gpu')
+__slots__ = ('request_memory',
+ 'request_cpu',
+ 'limit_memory',
+ 'limit_cpu',
+ 'limit_gpu',
+ 'request_ephemeral_storage',
+ 'limit_ephemeral_storage')
"""
:param request_memory: requested memory
@@ -44,15 +50,17 @@ class Resources(K8SModel):
:param limit_ephemeral_storage: Limit for ephemeral storage
:type limit_ephemeral_storage: float | str
"""
+
def __init__(
-self,
-request_memory=None,
-request_cpu=None,
-request_ephemeral_storage=None,
-limit_memory=None,
-limit_cpu=None,
-limit_gpu=None,
-limit_ephemeral_storage=None):
+self,
+request_memory=None,
+request_cpu=None,
+request_ephemeral_storage=None,
+limit_memory=None,
+limit_cpu=None,
+limit_gpu=None,
+limit_ephemeral_storage=None
+):
self.request_memory = request_memory
self.request_cpu = request_cpu
self.request_ephemeral_storage = request_ephemeral_storage
@@ -104,9 +112,10 @@ class Port(K8SModel):
__slots__ = ('name', 'container_port')
def __init__(
-self,
-name=None,
-container_port=None):
+self,
+name=None,
+container_port=None
+):
"""Creates port"""
self.name = name
self.container_port = container_port
diff --git a/airflow/kubernetes/pod_launcher.py
b/airflow/kubernetes/pod_launcher.py
index d27a647..05df204 100644
--- a/airflow/kubernetes/pod_launcher.py
+++ b/airflow/kubernetes/pod_launcher.py
@@ -26,10 +26,12 @@ from kubernetes.stream import stream as kubernetes_stream
from requests.exceptions import BaseHTTPError
from airflow import AirflowException
+from airflow.kubernetes.pod_launcher_helper import convert_to_airflow_pod
from airflow.kubernetes.pod_generator import PodDefaults
-from airflow.settings import pod_mutation_hook
+from airflow
[airflow] 22/32: Pin pymongo version to <3.11.0
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 5c9ff4d8176d4409dd19b5c16706d1936102279a Author: Kaxil Naik AuthorDate: Sun Aug 2 11:42:39 2020 +0100 Pin pymongo version to <3.11.0 --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index e11ce70..35323d2 100644 --- a/setup.py +++ b/setup.py @@ -307,7 +307,7 @@ ldap = [ ] mongo = [ 'dnspython>=1.13.0,<2.0.0', -'pymongo>=3.6.0', +'pymongo>=3.6.0,<3.11.0', ] mssql = [ 'pymssql~=2.1.1',
[airflow] 26/32: Allow to define custom XCom class (#8560)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 64c89db14308c567ac424201ed38cc452ddc6afd
Author: Tomek Urbaszek
AuthorDate: Tue Apr 28 16:55:05 2020 +0200
Allow to define custom XCom class (#8560)
* Allow to define custom XCom class
closes: #8059
(cherry picked from commit 6c6d6611d2aa112a947a9ebc7200446f51d0ac4c)
---
airflow/config_templates/config.yml | 7
airflow/config_templates/default_airflow.cfg | 4 +++
airflow/models/xcom.py | 34 ++-
docs/concepts.rst| 9 +
tests/models/test_xcom.py| 50
5 files changed, 103 insertions(+), 1 deletion(-)
diff --git a/airflow/config_templates/config.yml
b/airflow/config_templates/config.yml
index d1c2c90..f54255e 100644
--- a/airflow/config_templates/config.yml
+++ b/airflow/config_templates/config.yml
@@ -476,6 +476,13 @@
type: string
example: ~
default: "True"
+- name: xcom_backend
+ description: |
+Path to custom XCom class that will be used to store and resolve
operators results
+ version_added: 1.10.12
+ type: string
+ example: "path.to.CustomXCom"
+ default: "airflow.models.xcom.BaseXCom"
- name: secrets
description: ~
diff --git a/airflow/config_templates/default_airflow.cfg
b/airflow/config_templates/default_airflow.cfg
index bf83b34..e18e538 100644
--- a/airflow/config_templates/default_airflow.cfg
+++ b/airflow/config_templates/default_airflow.cfg
@@ -252,6 +252,10 @@ max_num_rendered_ti_fields_per_task = 30
# On each dagrun check against defined SLAs
check_slas = True
+# Path to custom XCom class that will be used to store and resolve operators
results
+# Example: xcom_backend = path.to.CustomXCom
+xcom_backend = airflow.models.xcom.BaseXCom
+
[secrets]
# Full class name of secrets backend to enable (will precede env vars and
metastore in search path)
# Example: backend =
airflow.contrib.secrets.aws_systems_manager.SystemsManagerParameterStoreBackend
diff --git a/airflow/models/xcom.py b/airflow/models/xcom.py
index f4522b5..0b6a81d 100644
--- a/airflow/models/xcom.py
+++ b/airflow/models/xcom.py
@@ -40,7 +40,7 @@ MAX_XCOM_SIZE = 49344
XCOM_RETURN_KEY = 'return_value'
-class XCom(Base, LoggingMixin):
+class BaseXCom(Base, LoggingMixin):
"""
Base class for XCom objects.
"""
@@ -232,3 +232,35 @@ class XCom(Base, LoggingMixin):
"for XCOM, then you need to enable pickle "
"support for XCOM in your airflow config.")
raise
+
+@staticmethod
+def deserialize_value(result) -> Any:
+# TODO: "pickling" has been deprecated and JSON is preferred.
+# "pickling" will be removed in Airflow 2.0.
+enable_pickling = conf.getboolean('core', 'enable_xcom_pickling')
+if enable_pickling:
+return pickle.loads(result.value)
+
+try:
+return json.loads(result.value.decode('UTF-8'))
+except ValueError:
+log.error("Could not deserialize the XCOM value from JSON. "
+ "If you are using pickles instead of JSON "
+ "for XCOM, then you need to enable pickle "
+ "support for XCOM in your airflow config.")
+raise
+
+
+def resolve_xcom_backend():
+"""Resolves custom XCom class"""
+clazz = conf.getimport("core", "xcom_backend",
fallback=f"airflow.models.xcom.{BaseXCom.__name__}")
+if clazz:
+if not issubclass(clazz, BaseXCom):
+raise TypeError(
+f"Your custom XCom class `{clazz.__name__}` is not a subclass
of `{BaseXCom.__name__}`."
+)
+return clazz
+return BaseXCom
+
+
+XCom = resolve_xcom_backend()
diff --git a/docs/concepts.rst b/docs/concepts.rst
index e85c5b3..dd48003 100644
--- a/docs/concepts.rst
+++ b/docs/concepts.rst
@@ -660,6 +660,15 @@ of what this may look like:
Note that XComs are similar to `Variables`_, but are specifically designed
for inter-task communication rather than global settings.
+Custom XCom backend
+'''
+
+It is possible to change ``XCom`` behaviour os serialization and
deserialization of tasks' result.
+To do this one have to change ``xcom_backend`` parameter in Airflow config.
Provided value should point
+to a class that is subclass of :class:`~airflow.models.xcom.BaseXCom`. To
alter the serialaization /
+deserialization mechanism the custom class should override ``serialize_value``
and ``deserialize_value``
+methods.
+
.. _concepts:variables:
Variables
diff --git a/tests/models/test_xcom.py b/tests/models/test_xcom.py
new file mode 100644
index 000..206b074
--- /dev/null
+++ b/tests/models/test_xcom.py
@@ -0,0 +1,50 @@
+# Licensed to the
[airflow] 30/32: Fix more PodMutationHook issues for backwards compatibility (#10084)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit c230156739178762d5cef482ace3d7a05e683cc1
Author: Kaxil Naik
AuthorDate: Fri Aug 7 11:50:44 2020 +0100
Fix more PodMutationHook issues for backwards compatibility (#10084)
Co-authored-by: Daniel Imberman
---
UPDATING.md | 10 +
airflow/contrib/executors/kubernetes_executor.py | 20 +
airflow/contrib/kubernetes/pod.py| 143 ++-
airflow/executors/kubernetes_executor.py | 6 +
airflow/kubernetes/pod.py| 31 +-
airflow/kubernetes/pod_generator.py | 76 +++-
airflow/kubernetes/pod_launcher.py | 73 +++-
airflow/kubernetes/pod_launcher_helper.py| 96 -
airflow/kubernetes/secret.py | 21 +-
airflow/kubernetes/volume.py | 17 +-
airflow/operators/python_operator.py | 4 +-
docs/conf.py | 1 +
kubernetes_tests/test_kubernetes_pod_operator.py | 1 -
tests/kubernetes/models/test_pod.py | 108 +++---
tests/kubernetes/models/test_volume.py | 40 ++
tests/kubernetes/test_pod_generator.py | 206 +-
tests/kubernetes/test_pod_launcher.py| 153 +++-
tests/kubernetes/test_pod_launcher_helper.py | 98 -
tests/kubernetes/test_worker_configuration.py| 7 +
tests/test_local_settings.py | 269 -
tests/test_local_settings/__init__.py| 16 +
tests/test_local_settings/test_local_settings.py | 461 +++
22 files changed, 1289 insertions(+), 568 deletions(-)
diff --git a/UPDATING.md b/UPDATING.md
index f82ba10..4f2b844 100644
--- a/UPDATING.md
+++ b/UPDATING.md
@@ -67,6 +67,16 @@ https://developers.google.com/style/inclusive-documentation
Previously, when tasks skipped by SkipMixin (such as BranchPythonOperator,
BaseBranchOperator and ShortCircuitOperator) are cleared, they execute. Since
1.10.12, when such skipped tasks are cleared,
they will be skipped again by the newly introduced NotPreviouslySkippedDep.
+### The pod_mutation_hook function will now accept a kubernetes V1Pod object
+
+As of airflow 1.10.12, using the `airflow.contrib.kubernetes.Pod` class in the
`pod_mutation_hook` is now deprecated. Instead we recommend that users
+treat the `pod` parameter as a `kubernetes.client.models.V1Pod` object. This
means that users now have access to the full Kubernetes API
+when modifying airflow pods
+
+### pod_template_file option now available in the KubernetesPodOperator
+
+Users can now offer a path to a yaml for the KubernetesPodOperator using the
`pod_template_file` parameter.
+
## Airflow 1.10.11
### Use NULL as default value for dag.description
diff --git a/airflow/contrib/executors/kubernetes_executor.py
b/airflow/contrib/executors/kubernetes_executor.py
new file mode 100644
index 000..416b2d7
--- /dev/null
+++ b/airflow/contrib/executors/kubernetes_executor.py
@@ -0,0 +1,20 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from airflow.executors import kubernetes_executor # noqa
diff --git a/airflow/contrib/kubernetes/pod.py
b/airflow/contrib/kubernetes/pod.py
index 0ab3616..944cd8c 100644
--- a/airflow/contrib/kubernetes/pod.py
+++ b/airflow/contrib/kubernetes/pod.py
@@ -19,7 +19,18 @@
import warnings
# pylint: disable=unused-import
-from airflow.kubernetes.pod import Port, Resources # noqa
+from typing import List, Union
+
+from kubernetes.client import models as k8s
+
+from airflow.kubernetes.pod import Port, Resources # noqa
+from airflow.kubernetes.volume import Volume
+from airflow.kubernetes.volume_mount import VolumeMount
+from airflow.kubernetes.secret import Secret
+
+from kubernetes.client.api_client import ApiClient
+
+api_client = ApiClient()
warnings.warn(
"This module is deprecated. Please use `airflow.kubernetes.pod`.",
@@ -120,7 +131,7 @@ class Pod(object):
self.affinity = affinity or {}
self.hostnetwork = hostnetwork or False
[airflow] 28/32: Add getimport for xcom change
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 1a8ba6a95696a19eb422a3d0a9553b3a59827052
Author: Daniel Imberman
AuthorDate: Mon Aug 3 15:59:40 2020 -0700
Add getimport for xcom change
---
airflow/configuration.py | 21 +
1 file changed, 21 insertions(+)
diff --git a/airflow/configuration.py b/airflow/configuration.py
index d912898..01ee90f 100644
--- a/airflow/configuration.py
+++ b/airflow/configuration.py
@@ -42,6 +42,7 @@ import yaml
from zope.deprecation import deprecated
from airflow.exceptions import AirflowConfigException
+from airflow.utils.module_loading import import_string
standard_library.install_aliases()
@@ -342,6 +343,26 @@ class AirflowConfigParser(ConfigParser):
"section/key [{section}/{key}] not found "
"in config".format(section=section, key=key))
+def getimport(self, section, key, **kwargs):
+"""
+Reads options, imports the full qualified name, and returns the object.
+In case of failure, it throws an exception a clear message with the
key aad the section names
+:return: The object or None, if the option is empty
+"""
+full_qualified_path = conf.get(section=section, key=key, **kwargs)
+if not full_qualified_path:
+return None
+
+try:
+return import_string(full_qualified_path)
+except ImportError as e:
+log.error(e)
+raise AirflowConfigException(
+'The object could not be loaded. Please check "{key}" key in
"{section}" section. '
+'Current value: "{full_qualified_path}".' .format(
+key=key, section=section,
full_qualified_path=full_qualified_path)
+)
+
def getboolean(self, section, key, **kwargs):
val = str(self.get(section, key, **kwargs)).lower().strip()
if '#' in val:
[airflow] 13/32: Simplify pull request template (#9896)
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 25e0e2617c536d4ac9128cbace381082441a3db0 Author: Tomek Urbaszek AuthorDate: Tue Jul 21 12:18:07 2020 +0200 Simplify pull request template (#9896) Remove the checklist of always checked points. (cherry picked from commit 7dd5c11f966df0cb20b7503be3438e19248e66ea) --- .github/PULL_REQUEST_TEMPLATE.md | 27 --- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 64625e4..1e3c23d 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,10 +1,23 @@ -- [ ] Description above provides context of the change -- [ ] Commit message contains [\[AIRFLOW-\]](https://issues.apache.org/jira/browse/AIRFLOW-) or `[AIRFLOW-]` for document-only changes -- [ ] Unit tests coverage for changes (not needed for documentation changes) -- [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)" -- [ ] Relevant documentation is updated including usage instructions. -- [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example). + + +--- +**^ Add meaningful description above** + +Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md). -Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
[airflow] 22/32: Pin pymongo version to <3.11.0
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 5c9ff4d8176d4409dd19b5c16706d1936102279a Author: Kaxil Naik AuthorDate: Sun Aug 2 11:42:39 2020 +0100 Pin pymongo version to <3.11.0 --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index e11ce70..35323d2 100644 --- a/setup.py +++ b/setup.py @@ -307,7 +307,7 @@ ldap = [ ] mongo = [ 'dnspython>=1.13.0,<2.0.0', -'pymongo>=3.6.0', +'pymongo>=3.6.0,<3.11.0', ] mssql = [ 'pymssql~=2.1.1',
[airflow] 11/32: Added "all" to allowed breeze integrations and tried to clarify on fail (#9872)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6e290cf0f850c36bb5386bf2f21045765bbbe600
Author: Alexander Sutcliffe <41974784+scrambldchan...@users.noreply.github.com>
AuthorDate: Sat Jul 18 07:38:02 2020 +0200
Added "all" to allowed breeze integrations and tried to clarify on fail
(#9872)
(cherry picked from commit 64929eeb70dcb5aa39ce3d504603721c20518cd4)
---
BREEZE.rst | 11 ++-
breeze | 1 -
breeze-complete| 2 +-
scripts/ci/pre_commit/pre_commit_check_integrations.sh | 4 +++-
4 files changed, 10 insertions(+), 8 deletions(-)
diff --git a/BREEZE.rst b/BREEZE.rst
index 435b21e..c377ec0 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -413,8 +413,10 @@ way as when you enter the environment. You can do it
multiple times and open as
CLIs for cloud providers
-Restarting Breeze environment
--
+For development convenience we installed simple wrappers for the most common
cloud providers CLIs. Those
+CLIs are not installed when you build or pull the image - they will be
downloaded as docker images
+the first time you attempt to use them. It is downloaded and executed in your
host's docker engine so once
+it is downloaded, it will stay until you remove the downloaded images from
your host container.
For each of those CLI credentials are taken (automatically) from the
credentials you have defined in
your ${HOME} directory on host.
@@ -697,7 +699,7 @@ Generating requirements
Whenever you modify and commit setup.py, you need to re-generate requirement
files. Those requirement
files ara stored separately for each python version in the ``requirements``
folder. Those are
constraints rather than requirements as described in detail in the
-`CONTRIBUTING `_ documentation.
+`CONTRIBUTING `_ contributing
documentation.
In case you modify setup.py you need to update the requirements - for every
python version supported.
@@ -1736,11 +1738,10 @@ This is the current syntax for `./breeze <./breeze>`_:
-i, --integration
Integration to start during tests - it determines which integrations
are started
for integration tests. There can be more than one integration
started, or all to
- }
start all integrations. Selected integrations are not saved for
future execution.
One of:
- cassandra kerberos mongo openldap rabbitmq redis
+ cassandra kerberos mongo openldap rabbitmq redis all
Kind kubernetes and Kubernetes tests configuration(optional)
diff --git a/breeze b/breeze
index abb95b7..571d383 100755
--- a/breeze
+++ b/breeze
@@ -1411,7 +1411,6 @@ function flag_breeze_actions() {
-i, --integration
Integration to start during tests - it determines which integrations
are started
for integration tests. There can be more than one integration started,
or all to
-}
start all integrations. Selected integrations are not saved for future
execution.
One of:
diff --git a/breeze-complete b/breeze-complete
index c1d955c..73368b8 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -19,7 +19,7 @@
_BREEZE_ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS="2.7 3.5 3.6 3.7 3.8"
_BREEZE_ALLOWED_BACKENDS="sqlite mysql postgres"
-_BREEZE_ALLOWED_INTEGRATIONS="cassandra kerberos mongo openldap rabbitmq redis"
+_BREEZE_ALLOWED_INTEGRATIONS="cassandra kerberos mongo openldap rabbitmq redis
all"
_BREEZE_ALLOWED_KUBERNETES_MODES="image git"
_BREEZE_ALLOWED_KUBERNETES_VERSIONS="v1.18.2"
_BREEZE_ALLOWED_HELM_VERSIONS="v3.2.4"
diff --git a/scripts/ci/pre_commit/pre_commit_check_integrations.sh
b/scripts/ci/pre_commit/pre_commit_check_integrations.sh
index 6871941..a0b33c8 100755
--- a/scripts/ci/pre_commit/pre_commit_check_integrations.sh
+++ b/scripts/ci/pre_commit/pre_commit_check_integrations.sh
@@ -26,7 +26,7 @@ cd "${AIRFLOW_SOURCES}" || exit 1
. breeze-complete
-if [[ ${AVAILABLE_INTEGRATIONS} != "${_BREEZE_ALLOWED_INTEGRATIONS}" ]]; then
+if [[ "${AVAILABLE_INTEGRATIONS} all" != "${_BREEZE_ALLOWED_INTEGRATIONS}" ]];
then
echo
echo "Error: Allowed integrations do not match!"
echo
@@ -36,6 +36,8 @@ if [[ ${AVAILABLE_INTEGRATIONS} !=
"${_BREEZE_ALLOWED_INTEGRATIONS}" ]]; then
echo "The ./breeze-complete integrations (_BREEZE_ALLOWED_INTEGRATIONS):"
echo "${_BREEZE_ALLOWED_INTEGRATIONS}"
echo
+ echo "_BREEZE_ALLOWED_INTEGRATIONS should match AVAILABLE_INTEGRATIONS plus
'all'"
+ echo
echo "Please align the two!"
echo
exit 1
[airflow] 02/32: Python base image version is retrieved in the right place (#9931)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e2e68538962a8b894314d4522056f8f39840a474
Author: Jarek Potiuk
AuthorDate: Wed Jul 22 15:58:39 2020 +0200
Python base image version is retrieved in the right place (#9931)
When quick-fixing Python 3.8.4 error #9820 PYTHON_BASE_IMAGE_VERSION
variable was added but it was initialized too early in Breeze and
it took the default version of Python rather than the one chosen
by --python switch. This caused the generated requirements
(locally by Breeze only) to generate wrong set of requirements
and images built locally for different python versions were
based on default Python version, not the one chosen by --python
switch.
(cherry picked from commit 7b9e8e0950f4b963b5cb13ed069d6132080fcd27)
---
breeze | 1 +
scripts/ci/libraries/_build_images.sh | 7 +++
scripts/ci/libraries/_initialization.sh | 4
3 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/breeze b/breeze
index 534bec3..a10ba9c 100755
--- a/breeze
+++ b/breeze
@@ -508,6 +508,7 @@ function prepare_command_files() {
export COMPOSE_CI_FILE
export COMPOSE_PROD_FILE
+get_base_image_version
# Base python image for the build
export PYTHON_BASE_IMAGE=python:${PYTHON_BASE_IMAGE_VERSION}-slim-buster
export
AIRFLOW_CI_IMAGE="${DOCKERHUB_USER}/${DOCKERHUB_REPO}:${BRANCH_NAME}-python${PYTHON_MAJOR_MINOR_VERSION}-ci"
diff --git a/scripts/ci/libraries/_build_images.sh
b/scripts/ci/libraries/_build_images.sh
index ed894a1..352975b 100644
--- a/scripts/ci/libraries/_build_images.sh
+++ b/scripts/ci/libraries/_build_images.sh
@@ -311,11 +311,17 @@ function print_build_info() {
print_info
}
+function get_base_image_version() {
+# python image version to use
+
PYTHON_BASE_IMAGE_VERSION=${PYTHON_BASE_IMAGE_VERSION:=${PYTHON_MAJOR_MINOR_VERSION}}
+}
+
# Prepares all variables needed by the CI build. Depending on the
configuration used (python version
# DockerHub user etc. the variables are set so that other functions can use
those variables.
function prepare_ci_build() {
+get_base_image_version
# We use pulled docker image cache by default for CI images to speed up
the builds
export DOCKER_CACHE=${DOCKER_CACHE:="pulled"}
echo
@@ -591,6 +597,7 @@ Docker building ${AIRFLOW_CI_IMAGE}.
# Prepares all variables needed by the CI build. Depending on the
configuration used (python version
# DockerHub user etc. the variables are set so that other functions can use
those variables.
function prepare_prod_build() {
+get_base_image_version
# We use local docker image cache by default for Production images
export DOCKER_CACHE=${DOCKER_CACHE:="local"}
echo
diff --git a/scripts/ci/libraries/_initialization.sh
b/scripts/ci/libraries/_initialization.sh
index c41dff9..5f2a742 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -21,10 +21,6 @@ function initialize_common_environment {
# default python Major/Minor version
PYTHON_MAJOR_MINOR_VERSION=${PYTHON_MAJOR_MINOR_VERSION:="3.6"}
-# python image version to use
-# shellcheck disable=SC2034
-PYTHON_BASE_IMAGE_VERSION=${PYTHON_MAJOR_MINOR_VERSION}
-
# extra flags passed to docker run for CI image
# shellcheck disable=SC2034
EXTRA_DOCKER_FLAGS=()
[airflow] 26/32: Allow to define custom XCom class (#8560)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 64c89db14308c567ac424201ed38cc452ddc6afd
Author: Tomek Urbaszek
AuthorDate: Tue Apr 28 16:55:05 2020 +0200
Allow to define custom XCom class (#8560)
* Allow to define custom XCom class
closes: #8059
(cherry picked from commit 6c6d6611d2aa112a947a9ebc7200446f51d0ac4c)
---
airflow/config_templates/config.yml | 7
airflow/config_templates/default_airflow.cfg | 4 +++
airflow/models/xcom.py | 34 ++-
docs/concepts.rst| 9 +
tests/models/test_xcom.py| 50
5 files changed, 103 insertions(+), 1 deletion(-)
diff --git a/airflow/config_templates/config.yml
b/airflow/config_templates/config.yml
index d1c2c90..f54255e 100644
--- a/airflow/config_templates/config.yml
+++ b/airflow/config_templates/config.yml
@@ -476,6 +476,13 @@
type: string
example: ~
default: "True"
+- name: xcom_backend
+ description: |
+Path to custom XCom class that will be used to store and resolve
operators results
+ version_added: 1.10.12
+ type: string
+ example: "path.to.CustomXCom"
+ default: "airflow.models.xcom.BaseXCom"
- name: secrets
description: ~
diff --git a/airflow/config_templates/default_airflow.cfg
b/airflow/config_templates/default_airflow.cfg
index bf83b34..e18e538 100644
--- a/airflow/config_templates/default_airflow.cfg
+++ b/airflow/config_templates/default_airflow.cfg
@@ -252,6 +252,10 @@ max_num_rendered_ti_fields_per_task = 30
# On each dagrun check against defined SLAs
check_slas = True
+# Path to custom XCom class that will be used to store and resolve operators
results
+# Example: xcom_backend = path.to.CustomXCom
+xcom_backend = airflow.models.xcom.BaseXCom
+
[secrets]
# Full class name of secrets backend to enable (will precede env vars and
metastore in search path)
# Example: backend =
airflow.contrib.secrets.aws_systems_manager.SystemsManagerParameterStoreBackend
diff --git a/airflow/models/xcom.py b/airflow/models/xcom.py
index f4522b5..0b6a81d 100644
--- a/airflow/models/xcom.py
+++ b/airflow/models/xcom.py
@@ -40,7 +40,7 @@ MAX_XCOM_SIZE = 49344
XCOM_RETURN_KEY = 'return_value'
-class XCom(Base, LoggingMixin):
+class BaseXCom(Base, LoggingMixin):
"""
Base class for XCom objects.
"""
@@ -232,3 +232,35 @@ class XCom(Base, LoggingMixin):
"for XCOM, then you need to enable pickle "
"support for XCOM in your airflow config.")
raise
+
+@staticmethod
+def deserialize_value(result) -> Any:
+# TODO: "pickling" has been deprecated and JSON is preferred.
+# "pickling" will be removed in Airflow 2.0.
+enable_pickling = conf.getboolean('core', 'enable_xcom_pickling')
+if enable_pickling:
+return pickle.loads(result.value)
+
+try:
+return json.loads(result.value.decode('UTF-8'))
+except ValueError:
+log.error("Could not deserialize the XCOM value from JSON. "
+ "If you are using pickles instead of JSON "
+ "for XCOM, then you need to enable pickle "
+ "support for XCOM in your airflow config.")
+raise
+
+
+def resolve_xcom_backend():
+"""Resolves custom XCom class"""
+clazz = conf.getimport("core", "xcom_backend",
fallback=f"airflow.models.xcom.{BaseXCom.__name__}")
+if clazz:
+if not issubclass(clazz, BaseXCom):
+raise TypeError(
+f"Your custom XCom class `{clazz.__name__}` is not a subclass
of `{BaseXCom.__name__}`."
+)
+return clazz
+return BaseXCom
+
+
+XCom = resolve_xcom_backend()
diff --git a/docs/concepts.rst b/docs/concepts.rst
index e85c5b3..dd48003 100644
--- a/docs/concepts.rst
+++ b/docs/concepts.rst
@@ -660,6 +660,15 @@ of what this may look like:
Note that XComs are similar to `Variables`_, but are specifically designed
for inter-task communication rather than global settings.
+Custom XCom backend
+'''
+
+It is possible to change ``XCom`` behaviour os serialization and
deserialization of tasks' result.
+To do this one have to change ``xcom_backend`` parameter in Airflow config.
Provided value should point
+to a class that is subclass of :class:`~airflow.models.xcom.BaseXCom`. To
alter the serialaization /
+deserialization mechanism the custom class should override ``serialize_value``
and ``deserialize_value``
+methods.
+
.. _concepts:variables:
Variables
diff --git a/tests/models/test_xcom.py b/tests/models/test_xcom.py
new file mode 100644
index 000..206b074
--- /dev/null
+++ b/tests/models/test_xcom.py
@@ -0,0 +1,50 @@
+# Licensed to the
[airflow] 27/32: Make XCom 2.7 compatible
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit ec1cb7da1464708ba83bbfe0bc5efc367ea361c3
Author: Daniel Imberman
AuthorDate: Mon Aug 3 14:23:18 2020 -0700
Make XCom 2.7 compatible
---
airflow/models/xcom.py | 8 +---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/airflow/models/xcom.py b/airflow/models/xcom.py
index 0b6a81d..9861c2c 100644
--- a/airflow/models/xcom.py
+++ b/airflow/models/xcom.py
@@ -234,7 +234,7 @@ class BaseXCom(Base, LoggingMixin):
raise
@staticmethod
-def deserialize_value(result) -> Any:
+def deserialize_value(result):
# TODO: "pickling" has been deprecated and JSON is preferred.
# "pickling" will be removed in Airflow 2.0.
enable_pickling = conf.getboolean('core', 'enable_xcom_pickling')
@@ -253,11 +253,13 @@ class BaseXCom(Base, LoggingMixin):
def resolve_xcom_backend():
"""Resolves custom XCom class"""
-clazz = conf.getimport("core", "xcom_backend",
fallback=f"airflow.models.xcom.{BaseXCom.__name__}")
+clazz = conf.getimport("core", "xcom_backend",
fallback="airflow.models.xcom.{}"
+ .format(BaseXCom.__name__))
if clazz:
if not issubclass(clazz, BaseXCom):
raise TypeError(
-f"Your custom XCom class `{clazz.__name__}` is not a subclass
of `{BaseXCom.__name__}`."
+"Your custom XCom class `{class_name}` is not a subclass of
`{base_name}`."
+.format(class_name=clazz.__name__, base_name=BaseXCom.__name__)
)
return clazz
return BaseXCom
[airflow] 29/32: Pin fsspec<8.0.0 for Python <3.6 to fix Static Checks
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 3e1d88eb2aeaca5103cd7dce536750bd48a6c37f Author: Kaxil Naik AuthorDate: Thu Aug 6 20:19:35 2020 +0100 Pin fsspec<8.0.0 for Python <3.6 to fix Static Checks --- setup.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 327e157..e16b1cd 100644 --- a/setup.py +++ b/setup.py @@ -325,7 +325,8 @@ pagerduty = [ papermill = [ 'papermill[all]>=1.0.0', 'nteract-scrapbook[all]>=0.2.1', -'pyarrow<1.0.0' +'pyarrow<1.0.0', +'fsspec<0.8.0;python_version=="3.5"' ] password = [ 'bcrypt>=2.0.0',
[airflow] 03/32: Fix task_instance_mutation_hook (#9910)
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit e6b017aff28ef2f7c08998386e8221abe1b553de Author: Jarek Potiuk AuthorDate: Wed Jul 22 16:55:54 2020 +0200 Fix task_instance_mutation_hook (#9910) Fixes #9902 --- airflow/__init__.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/airflow/__init__.py b/airflow/__init__.py index c8dcd21..287efe3 100644 --- a/airflow/__init__.py +++ b/airflow/__init__.py @@ -40,12 +40,14 @@ import sys from airflow import utils from airflow import settings from airflow.configuration import conf -from airflow.models import DAG from flask_admin import BaseView from importlib import import_module from airflow.exceptions import AirflowException settings.initialize() +# Delay the import of airflow.models to be after the settings initialization to make sure that +# any reference to a settings' functions (e.g task_instance_mutation_hook) holds the expected implementation +from airflow.models import DAG # noqa: E402 login = None # type: Any log = logging.getLogger(__name__)
[airflow] 19/32: Fix bug in executor_config when defining resources (#9935)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 05ec21a22f84cdbe2aaed38b712c30f2cbb38b59
Author: Daniel Imberman
AuthorDate: Thu Jul 23 19:52:20 2020 -0700
Fix bug in executor_config when defining resources (#9935)
* Fix PodGenerator to handle Kubernetes resources
In Airflow 1.10.11, `namespaced['resources'] = resources` is missing.
This PR improves the definition of pod resources, `requests` and `limits`
are optional.
* Make it working in 2.7
* Add limit_gpu and fix ephemeral-storage keys
* Fix flake8
Co-authored-by: Riccardo Bini
---
airflow/kubernetes/pod.py | 4 +-
airflow/kubernetes/pod_generator.py| 33 ++---
tests/kubernetes/test_pod_generator.py | 132 +
3 files changed, 155 insertions(+), 14 deletions(-)
diff --git a/airflow/kubernetes/pod.py b/airflow/kubernetes/pod.py
index b1df462..0b332c2 100644
--- a/airflow/kubernetes/pod.py
+++ b/airflow/kubernetes/pod.py
@@ -33,7 +33,7 @@ class Resources(K8SModel):
:type request_memory: str
:param request_cpu: requested CPU number
:type request_cpu: float | str
-:param request_ephemeral_storage: requested ephermeral storage
+:param request_ephemeral_storage: requested ephemeral storage
:type request_ephemeral_storage: str
:param limit_memory: limit for memory usage
:type limit_memory: str
@@ -41,7 +41,7 @@ class Resources(K8SModel):
:type limit_cpu: float | str
:param limit_gpu: Limits for GPU used
:type limit_gpu: int
-:param limit_ephemeral_storage: Limit for ephermeral storage
+:param limit_ephemeral_storage: Limit for ephemeral storage
:type limit_ephemeral_storage: float | str
"""
def __init__(
diff --git a/airflow/kubernetes/pod_generator.py
b/airflow/kubernetes/pod_generator.py
index e46407b..d11c175 100644
--- a/airflow/kubernetes/pod_generator.py
+++ b/airflow/kubernetes/pod_generator.py
@@ -344,18 +344,26 @@ class PodGenerator(object):
resources = namespaced.get('resources')
if resources is None:
-requests = {
-'cpu': namespaced.get('request_cpu'),
-'memory': namespaced.get('request_memory'),
-'ephemeral-storage': namespaced.get('ephemeral-storage')
-}
-limits = {
-'cpu': namespaced.get('limit_cpu'),
-'memory': namespaced.get('limit_memory'),
-'ephemeral-storage': namespaced.get('ephemeral-storage')
-}
-all_resources = list(requests.values()) + list(limits.values())
-if all(r is None for r in all_resources):
+def extract(cpu, memory, ephemeral_storage, limit_gpu=None):
+resources_obj = {
+'cpu': namespaced.pop(cpu, None),
+'memory': namespaced.pop(memory, None),
+'ephemeral-storage': namespaced.pop(ephemeral_storage,
None),
+}
+if limit_gpu is not None:
+resources_obj['nvidia.com/gpu'] =
namespaced.pop(limit_gpu, None)
+
+resources_obj = {k: v for k, v in resources_obj.items() if v
is not None}
+
+if all(r is None for r in resources_obj):
+resources_obj = None
+return namespaced, resources_obj
+
+namespaced, requests = extract('request_cpu', 'request_memory',
'request_ephemeral_storage')
+namespaced, limits = extract('limit_cpu', 'limit_memory',
'limit_ephemeral_storage',
+ limit_gpu='limit_gpu')
+
+if requests is None and limits is None:
resources = None
else:
resources = k8s.V1ResourceRequirements(
@@ -371,6 +379,7 @@ class PodGenerator(object):
'iam.cloud.google.com/service-account': gcp_service_account_key
})
+namespaced['resources'] = resources
return PodGenerator(**namespaced).gen_pod()
@staticmethod
diff --git a/tests/kubernetes/test_pod_generator.py
b/tests/kubernetes/test_pod_generator.py
index 7d39cdc..d0faf4c 100644
--- a/tests/kubernetes/test_pod_generator.py
+++ b/tests/kubernetes/test_pod_generator.py
@@ -288,6 +288,138 @@ class TestPodGenerator(unittest.TestCase):
}, result)
@mock.patch('uuid.uuid4')
+def test_from_obj_with_resources(self, mock_uuid):
+self.maxDiff = None
+
+mock_uuid.return_value = self.static_uuid
+result = PodGenerator.from_obj({
+"KubernetesExecutor": {
+"annotations": {"test": "annotation"},
+"volumes": [
+{
+"name": "example-kubernetes-test-volume",
+
[airflow] 17/32: Pin google-cloud-container to <2 (#9901)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6d10a7f71c6aa10f7020bd4b69355f36bc9179b7
Author: Ephraim Anierobi
AuthorDate: Mon Jul 20 22:56:37 2020 +0100
Pin google-cloud-container to <2 (#9901)
(cherry picked from commit 560e0b504b52ead405b604934893c784ddf4dafa)
---
Dockerfile.ci | 2 +-
setup.py | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 5d4f240..2797e34 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -186,7 +186,7 @@ RUN mkdir -pv ${AIRFLOW_HOME} && \
mkdir -pv ${AIRFLOW_HOME}/logs
# Increase the value here to force reinstalling Apache Airflow pip dependencies
-ARG PIP_DEPENDENCIES_EPOCH_NUMBER="3"
+ARG PIP_DEPENDENCIES_EPOCH_NUMBER="4"
ENV PIP_DEPENDENCIES_EPOCH_NUMBER=${PIP_DEPENDENCIES_EPOCH_NUMBER}
# Optimizing installation of Cassandra driver
diff --git a/setup.py b/setup.py
index 1fe821b..e11ce70 100644
--- a/setup.py
+++ b/setup.py
@@ -255,7 +255,7 @@ gcp = [
'google-auth>=1.0.0, <2.0.0dev',
'google-auth-httplib2>=0.0.1',
'google-cloud-bigtable>=1.0.0',
-'google-cloud-container>=0.1.1',
+'google-cloud-container>=0.1.1,<2.0',
'google-cloud-dlp>=0.11.0',
'google-cloud-language>=1.1.1',
'google-cloud-secret-manager>=0.2.0',
[airflow] 05/32: Tests should also be triggered when there is just setup.py change (#9690)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 0718977d8b15754eec0a9bafbd8a44e556b1d5bb
Author: Jarek Potiuk
AuthorDate: Mon Jul 6 20:41:35 2020 +0200
Tests should also be triggered when there is just setup.py change (#9690)
So far tests were not triggered when only requirements changed,
but this is quite needed in fact.
(cherry picked from commit 72abf824cef6a1d82ecf882756206f02ed6a6864)
---
.github/workflows/ci.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 029c341..134bc1f 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -113,7 +113,7 @@ jobs:
run: |
set +e
./scripts/ci/ci_count_changed_files.sh ${GITHUB_SHA} \
- '^airflow|.github/workflows/|^Dockerfile|^scripts|^chart'
+
'^airflow|.github/workflows/|^Dockerfile|^scripts|^chart|^setup.py|^requirements|^tests|^kubernetes_tests'
echo "::set-output name=count::$?"
id: trigger-tests
[airflow] branch v1-10-test updated (a281faa -> 242d6d0)
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a change to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git. discard a281faa fix static tests discard 71c6ace Makes multi-namespace mode optional (#9570) discard 14ca77d Fix KubernetesPodOperator reattachment (#10230) omit 90fe915 add documentation for k8s fixes omit 9e2557a Fix init containers omit 215c16f Init container as dict instead of object omit 3773dac init-container fix omit be06b18 Add kubernetes changes to UPDATING.md omit 33626c1 Fix volume secret extraction omit 9b89f07 Allows secrets with mounts in init containers omit 21aade4 Fix more PodMutationHook issues for backwards compatibility (#10084) omit a513964 fixup! Pin fsspec<8.0.0 for Python <3.6 to fix Static Checks omit badd86a Pin fsspec<8.0.0 for Python <3.6 to fix Static Checks omit 54d8aae Revert "Enable pretty output in mypy (#9785)" omit fa88bf9 compat fix for Xcom omit aab7b59 Add getimport for xcom change omit ca0dd1a Make XCom 2.7 compatible omit 5e3ba14 Allow to define custom XCom class (#8560) omit 707f995 Fix check_integration pre-commit test (#9869) omit ab1d7ec Enable pretty output in mypy (#9785) omit f872c54 Fix docstrings in BigQueryGetDataOperator (#10042) omit e8d6db6 Pin Pyarrow < 1.0 omit 2cb1d82 Set pytest version to be < 6.0.0 due to breaking changes (#10043) omit f83560d Pin pymongo version to <3.11.0 omit 4f7a453 Add pre 1.10.11 Kubernetes Paths back with Deprecation Warning (#10067) omit ea6c305 Fixes PodMutationHook for backwards compatibility (#9903) omit 133285f Fix bug in executor_config when defining resources (#9935) omit 8c6bebe Breeze / KinD - support earlier k8s versions, fix recreate and kubectl versioning (#9905) omit 5382db3 Pin google-cloud-container to <2 (#9901) omit 4f8b838 Clean up tmp directory when exiting from breeze shell (#9930) omit 7d24e73 Pin github checkout action to v2 (#9938) omit babe75c fixup! fixup! Constraint files are now maintained automatically (#9889) omit 15fc941 fixup! Constraint files are now maintained automatically (#9889) omit 627aa71 Tests are cancelled if any of faster checks fail (#9917) omit 5b03424 Simplify pull request template (#9896) omit 26d6c6e Constraint files are now maintained automatically (#9889) omit 86cc96c Added "all" to allowed breeze integrations and tried to clarify on fail (#9872) omit d33f6d0 Reorganizing of CI tests (#9654) omit 07d0c84 Group CI scripts in subdirectories (#9653) omit dff4443 For now cloud tools are not needed in CI (#9818) omit d66494f Remove package.json and yarn.lock from the prod image (#9814) omit 5f4f8f4 The group of embedded DAGs should be root to be OpenShift compatible (#9794) omit 1c51f98 Tests should also be triggered when there is just setup.py change (#9690) omit 6051b41 [AIRFLOW-5391] Do not re-run skipped tasks when they are cleared (#7276) omit 320ccad Fix task_instance_mutation_hook (#9910) omit 8ea8dd1 fixup! Update some dependencies (#9684) omit 29835cd Python base image version is retrieved in the right place (#9931) omit 0dc20fd Update some dependencies (#9684) new 0b5f0fc Update some dependencies (#9684) new e2e6853 Python base image version is retrieved in the right place (#9931) new e6b017a Fix task_instance_mutation_hook (#9910) new 179e930 [AIRFLOW-5391] Do not re-run skipped tasks when they are cleared (#7276) new 0718977 Tests should also be triggered when there is just setup.py change (#9690) new d61c33d The group of embedded DAGs should be root to be OpenShift compatible (#9794) new 1d4782e Remove package.json and yarn.lock from the prod image (#9814) new 1a41879 For now cloud tools are not needed in CI (#9818) new 7ec2b3a Group CI scripts in subdirectories (#9653) new f6c8f51 Reorganizing of CI tests (#9654) new 6e290cf Added "all" to allowed breeze integrations and tried to clarify on fail (#9872) new 5f93baf Constraint files are now maintained automatically (#9889) new 25e0e26 Simplify pull request template (#9896) new e33ffbe Tests are cancelled if any of faster checks fail (#9917) new c72ce92 Pin github checkout action to v2 (#9938) new 213500e Clean up tmp directory when exiting from breeze shell (#9930) new 6d10a7f Pin google-cloud-container to <2 (#9901) new 6b219e1 Breeze / KinD - support earlier k8s versions, fix recreate and kubectl versioning (#9905) new 05ec21a Fix bug in executor_config when defining resources (#9935) new bcd02dd Fixes PodMutationHook for backwards compatibility (#9903) new bfa089d Add pre 1.10.11 Kubernetes Paths back with Deprecation Warning (#10067) new 5c9ff4d Pin pymongo
[airflow] 14/32: Tests are cancelled if any of faster checks fail (#9917)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit e33ffbef7db99a220642d8b12b0a4c25275ef598
Author: Jarek Potiuk
AuthorDate: Wed Jul 22 08:05:36 2020 +0200
Tests are cancelled if any of faster checks fail (#9917)
(cherry picked from commit 508d7d202ac13531ac840e8fdcc1ad0adb7c9460)
---
.github/workflows/ci.yml | 14 +-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index aac8be1..8cb1efa 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -60,7 +60,7 @@ jobs:
static-checks:
timeout-minutes: 60
-name: "Checks"
+name: "Static checks"
runs-on: ubuntu-latest
needs:
- cancel-previous-workflow-run
@@ -84,6 +84,9 @@ jobs:
run: ./scripts/ci/images/ci_prepare_ci_image_on_ci.sh
- name: "Static checks"
run: ./scripts/ci/static_checks/ci_run_static_checks.sh
+ - name: "Cancel workflow on static checks failure"
+if: ${{ failure() }}
+uses: andymckay/cancel-action@0.2
docs:
timeout-minutes: 60
name: "Build docs"
@@ -99,6 +102,9 @@ jobs:
run: ./scripts/ci/images/ci_prepare_ci_image_on_ci.sh
- name: "Build docs"
run: ./scripts/ci/docs/ci_docs.sh
+ - name: "Cancel workflow on docs failure"
+if: ${{ failure() }}
+uses: andymckay/cancel-action@0.2
trigger-tests:
timeout-minutes: 5
@@ -305,6 +311,9 @@ jobs:
- uses: actions/checkout@master
- name: "Helm Tests"
run: ./scripts/ci/kubernetes/ci_run_helm_testing.sh
+ - name: "Cancel workflow on helm-tests failure"
+if: ${{ failure() }}
+uses: andymckay/cancel-action@0.2
build-prod-image:
timeout-minutes: 60
@@ -319,6 +328,9 @@ jobs:
- uses: actions/checkout@master
- name: "Build PROD image ${{ matrix.python-version }}"
run: ./scripts/ci/images/ci_prepare_prod_image_on_ci.sh
+ - name: "Cancel workflow on build prod image failure"
+if: ${{ failure() }}
+uses: andymckay/cancel-action@0.2
push-prod-images-to-github-cache:
timeout-minutes: 80
[airflow] 23/32: Set pytest version to be < 6.0.0 due to breaking changes (#10043)
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 6f8b0ccd88a8f69b7f6efbc68adee15791daae72 Author: Felix Uellendall AuthorDate: Wed Jul 29 10:45:34 2020 +0200 Set pytest version to be < 6.0.0 due to breaking changes (#10043) The latest pytest version 6.0.0 released yesterday (2020-07-28) does not work in conjunction with the version of pylint (2.4.3) we are using. (cherry picked from commit 2e0d91d8eb9bcae1886358791917b953330a957f) --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 35323d2..cc0f721 100644 --- a/setup.py +++ b/setup.py @@ -422,7 +422,7 @@ devel = [ 'paramiko', 'pre-commit', 'pysftp', -'pytest', +'pytest<6.0.0', # FIXME: pylint complaining for pytest.mark.* on v6.0 'pytest-cov', 'pytest-instafail', 'pytest-rerunfailures',
[airflow] 10/32: Reorganizing of CI tests (#9654)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit f6c8f51d982ae50c8524a740b9e44a974d5c8748
Author: Jarek Potiuk
AuthorDate: Fri Jul 17 10:30:56 2020 +0200
Reorganizing of CI tests (#9654)
* we come back to idea of having one CI workflow
* cancel and openapi are incorporated into that CI workflow
* cancel retrieves workflow id automatically (works for forks)
* static checks are now merged into one job
* less dependencies between jobs so that waiting is minimised
* better name for check if tests should be run
* separated out script for tests should be run check
(cherry picked from commit 496ed6f1b279dc5ca560dec0044c8373531565f4)
---
.github/workflows/ci.yml | 102 -
.../get_workflow_id.sh}| 18 ++--
scripts/ci/{ => kubernetes}/ci_run_helm_testing.sh | 2 +-
scripts/ci/libraries/_initialization.sh| 2 +-
...files.sh => ci_check_if_tests_should_be_run.sh} | 69 +++---
scripts/ci/tools/ci_count_changed_files.sh | 19 +++-
6 files changed, 119 insertions(+), 93 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index a849d58..604fa0d 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -40,10 +40,30 @@ env:
jobs:
+ cancel-previous-workflow-run:
+timeout-minutes: 60
+name: "Cancel previous workflow run"
+runs-on: ubuntu-latest
+steps:
+ - uses: actions/checkout@master
+ - name: Get ci workflow id
+run: "scripts/ci/cancel/get_workflow_id.sh"
+env:
+ WORKFLOW: ci
+ GITHUB_TOKEN: ${{ github.token }}
+ GITHUB_REPOSITORY: ${{ github.repositoru }}
+ - name: Cancel workflow ${{ github.workflow }}
+uses: styfle/cancel-workflow-action@0.3.2
+with:
+ workflow_id: ${{ env.WORKFLOW_ID }}
+ access_token: ${{ github.token }}
+
static-checks:
timeout-minutes: 60
name: "Checks"
runs-on: ubuntu-latest
+needs:
+ - cancel-previous-workflow-run
env:
MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
CI_JOB_TYPE: "Static checks"
@@ -69,11 +89,12 @@ jobs:
python -m pip install pre-commit \
--constraint
requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt
./scripts/ci/static_checks/ci_run_static_checks.sh
-
docs:
timeout-minutes: 60
-name: Build docs
+name: "Build docs"
runs-on: ubuntu-latest
+needs:
+ - cancel-previous-workflow-run
env:
CI_JOB_TYPE: "Documentation"
steps:
@@ -86,42 +107,25 @@ jobs:
- name: "Build docs"
run: ./scripts/ci/docs/ci_docs.sh
- build-prod-image:
-timeout-minutes: 60
-name: "Build prod image Py${{ matrix.python-version }}"
-runs-on: ubuntu-latest
-strategy:
- matrix:
-python-version: [2.7, 3.5, 3.6, 3.7, 3.8]
-env:
- PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
- CI_JOB_TYPE: "Prod image"
-steps:
- - uses: actions/checkout@master
- - name: "Build PROD image ${{ matrix.python-version }}"
-run: ./scripts/ci/images/ci_prepare_prod_image_on_ci.sh
-
trigger-tests:
-timeout-minutes: 10
-name: "Count changed important files"
+timeout-minutes: 5
+name: "Checks if tests should be run"
runs-on: ubuntu-latest
+needs:
+ - cancel-previous-workflow-run
outputs:
- count: ${{ steps.trigger-tests.outputs.count }}
+ run-tests: ${{ steps.trigger-tests.outputs.run-tests }}
steps:
- uses: actions/checkout@master
- - name: "Get count of changed python files"
-run: |
- set +e
- ./scripts/ci/tools/ci_count_changed_files.sh ${GITHUB_SHA} \
-
'^airflow|.github/workflows/|^Dockerfile|^scripts|^chart|^setup.py|^requirements|^tests|^kubernetes_tests'
- echo "::set-output name=count::$?"
+ - name: "Check if tests should be run"
+run: "./scripts/ci/tools/ci_check_if_tests_should_be_run.sh"
id: trigger-tests
tests-kubernetes:
timeout-minutes: 80
name: "K8s: ${{matrix.kube-mode}} ${{matrix.python-version}}
${{matrix.kubernetes-version}}"
runs-on: ubuntu-latest
-needs: [static-checks, trigger-tests]
+needs: [trigger-tests]
strategy:
matrix:
python-version: [3.6, 3.7]
@@ -147,8 +151,7 @@ jobs:
KUBERNETES_VERSION: "${{ matrix.kubernetes-version }}"
KIND_VERSION: "${{ matrix.kind-version }}"
HELM_VERSION: "${{ matrix.helm-version }}"
-# For pull requests only run tests when important files changed
-if: needs.trigger-tests.outputs.count != '0' || github.event_name !=
'pull_request'
+if: needs.trigger-tests.outputs.run-tests == 'true' || github.event_name
[airflow] 15/32: Pin github checkout action to v2 (#9938)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit c72ce92d54aa9011187de62b0caa7b2f970034f0
Author: Jarek Potiuk
AuthorDate: Wed Jul 22 21:18:17 2020 +0200
Pin github checkout action to v2 (#9938)
(cherry picked from commit e86d753b4ba83dd3f27c613d48493c088abaa2b8)
---
.github/workflows/ci.yml | 28 ++--
1 file changed, 14 insertions(+), 14 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 8cb1efa..d0e7ab2 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -45,7 +45,7 @@ jobs:
name: "Cancel previous workflow run"
runs-on: ubuntu-latest
steps:
- - uses: actions/checkout@master
+ - uses: actions/checkout@v2
- name: Get ci workflow id
run: "scripts/ci/cancel/get_workflow_id.sh"
env:
@@ -67,7 +67,7 @@ jobs:
env:
MOUNT_SOURCE_DIR_FOR_STATIC_CHECKS: "true"
steps:
- - uses: actions/checkout@master
+ - uses: actions/checkout@v2
- uses: actions/setup-python@v1
with:
python-version: '3.x'
@@ -94,7 +94,7 @@ jobs:
needs:
- cancel-previous-workflow-run
steps:
- - uses: actions/checkout@master
+ - uses: actions/checkout@v2
- uses: actions/setup-python@v1
with:
python-version: '3.6'
@@ -115,7 +115,7 @@ jobs:
outputs:
run-tests: ${{ steps.trigger-tests.outputs.run-tests }}
steps:
- - uses: actions/checkout@master
+ - uses: actions/checkout@v2
- name: "Check if tests should be run"
run: "./scripts/ci/tools/ci_check_if_tests_should_be_run.sh"
id: trigger-tests
@@ -151,7 +151,7 @@ jobs:
HELM_VERSION: "${{ matrix.helm-version }}"
if: needs.trigger-tests.outputs.run-tests == 'true' || github.event_name
!= 'pull_request'
steps:
- - uses: actions/checkout@master
+ - uses: actions/checkout@v2
- uses: actions/setup-python@v1
with:
python-version: '3.6'
@@ -201,7 +201,7 @@ jobs:
TEST_TYPE: ${{ matrix.test-type }}
if: needs.trigger-tests.outputs.run-tests == 'true' || github.event_name
!= 'pull_request'
steps:
- - uses: actions/checkout@master
+ - uses: actions/checkout@v2
- uses: actions/setup-python@v1
with:
python-version: '3.6'
@@ -231,7 +231,7 @@ jobs:
TEST_TYPE: ${{ matrix.test-type }}
if: needs.trigger-tests.outputs.run-tests == 'true' || github.event_name
!= 'pull_request'
steps:
- - uses: actions/checkout@master
+ - uses: actions/checkout@v2
- uses: actions/setup-python@v1
with:
python-version: '3.x'
@@ -259,7 +259,7 @@ jobs:
RUN_TESTS: "true"
if: needs.trigger-tests.outputs.run-tests == 'true' || github.event_name
!= 'pull_request'
steps:
- - uses: actions/checkout@master
+ - uses: actions/checkout@v2
- uses: actions/setup-python@v1
with:
python-version: '3.x'
@@ -290,7 +290,7 @@ jobs:
TEST_TYPE: ${{ matrix.test-type }}
if: needs.trigger-tests.outputs.run-tests == 'true' || github.event_name
!= 'pull_request'
steps:
- - uses: actions/checkout@master
+ - uses: actions/checkout@v2
- uses: actions/setup-python@v1
with:
python-version: '3.x'
@@ -308,7 +308,7 @@ jobs:
needs:
- cancel-previous-workflow-run
steps:
- - uses: actions/checkout@master
+ - uses: actions/checkout@v2
- name: "Helm Tests"
run: ./scripts/ci/kubernetes/ci_run_helm_testing.sh
- name: "Cancel workflow on helm-tests failure"
@@ -325,7 +325,7 @@ jobs:
env:
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
steps:
- - uses: actions/checkout@master
+ - uses: actions/checkout@v2
- name: "Build PROD image ${{ matrix.python-version }}"
run: ./scripts/ci/images/ci_prepare_prod_image_on_ci.sh
- name: "Cancel workflow on build prod image failure"
@@ -351,7 +351,7 @@ jobs:
env:
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
steps:
- - uses: actions/checkout@master
+ - uses: actions/checkout@v2
- name: "Free space"
run: ./scripts/ci/tools/ci_free_space_on_ci.sh
- name: "Build PROD images ${{ matrix.python-version }}"
@@ -378,7 +378,7 @@ jobs:
PULL_PYTHON_BASE_IMAGES_FROM_CACHE: "false"
PYTHON_MAJOR_MINOR_VERSION: ${{ matrix.python-version }}
steps:
- - uses: actions/checkout@master
+ - uses: actions/checkout@v2
- name: "Free space"
run: ./scripts/ci/tools/ci_free_space_on_ci.sh
- name: "Build CI image"
@@ -406,7 +406,7 @@ jobs:
github.event_name == 'push' &&
(github.ref == 'refs/heads/master' || github.ref ==
'refs/heads/v1-10-test')
[airflow] 18/32: Breeze / KinD - support earlier k8s versions, fix recreate and kubectl versioning (#9905)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6b219e19f32ccb7017fae33a761fa5e800f6e56f
Author: Beni Ben zikry
AuthorDate: Wed Jul 22 13:24:46 2020 +0300
Breeze / KinD - support earlier k8s versions, fix recreate and kubectl
versioning (#9905)
(cherry picked from commit 24a951e8ed7159d3c2cbdb76d8d4ab807ee95cbd)
---
.github/workflows/ci.yml| 3 +--
BREEZE.rst | 4 ++--
breeze | 4 +++-
breeze-complete | 2 +-
scripts/ci/libraries/_initialization.sh | 2 +-
scripts/ci/libraries/_kind.sh | 6 +++---
6 files changed, 11 insertions(+), 10 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index d0e7ab2..72f6ba5 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -130,8 +130,7 @@ jobs:
python-version: [3.6, 3.7]
kube-mode:
- image
-kubernetes-version:
- - "v1.18.2"
+kubernetes-version: [v1.18.6, v1.17.5, v1.16.9]
kind-version:
- "v0.8.0"
helm-version:
diff --git a/BREEZE.rst b/BREEZE.rst
index 5976b0f..60ea13a 100644
--- a/BREEZE.rst
+++ b/BREEZE.rst
@@ -1756,9 +1756,9 @@ This is the current syntax for `./breeze <./breeze>`_:
Kubernetes version - only used in case one of --kind-cluster-*
commands is used.
One of:
- v1.18.2
+ v1.18.6 v1.17.5 v1.16.9
- Default: v1.18.2
+ Default: v1.18.6
--kind-version
Kind version - only used in case one of --kind-cluster-* commands is
used.
diff --git a/breeze b/breeze
index 91585a1..44e8d03 100755
--- a/breeze
+++ b/breeze
@@ -146,7 +146,7 @@ function setup_default_breeze_variables() {
_BREEZE_DEFAULT_BACKEND="sqlite"
_BREEZE_DEFAULT_KUBERNETES_MODE="image"
-_BREEZE_DEFAULT_KUBERNETES_VERSION="v1.18.2"
+_BREEZE_DEFAULT_KUBERNETES_VERSION="v1.18.6"
_BREEZE_DEFAULT_KIND_VERSION="v0.8.0"
_BREEZE_DEFAULT_HELM_VERSION="v3.2.4"
_BREEZE_DEFAULT_POSTGRES_VERSION="9.6"
@@ -1927,6 +1927,8 @@ function run_build_command {
echo "Stops KinD cluster"
elif [[ ${KIND_CLUSTER_OPERATION} == "restart" ]] ; then
echo "Restarts KinD cluster"
+elif [[ ${KIND_CLUSTER_OPERATION} == "recreate" ]] ; then
+ echo "Recreates KinD cluster"
elif [[ ${KIND_CLUSTER_OPERATION} == "status" ]] ; then
echo "Checks status of KinD cluster"
elif [[ ${KIND_CLUSTER_OPERATION} == "deploy" ]] ; then
diff --git a/breeze-complete b/breeze-complete
index 574b9c7..45732b3 100644
--- a/breeze-complete
+++ b/breeze-complete
@@ -21,7 +21,7 @@ _BREEZE_ALLOWED_PYTHON_MAJOR_MINOR_VERSIONS="2.7 3.5 3.6 3.7
3.8"
_BREEZE_ALLOWED_BACKENDS="sqlite mysql postgres"
_BREEZE_ALLOWED_INTEGRATIONS="cassandra kerberos mongo openldap rabbitmq redis
all"
_BREEZE_ALLOWED_KUBERNETES_MODES="image git"
-_BREEZE_ALLOWED_KUBERNETES_VERSIONS="v1.18.2"
+_BREEZE_ALLOWED_KUBERNETES_VERSIONS="v1.18.6 v1.17.5 v1.16.9"
_BREEZE_ALLOWED_HELM_VERSIONS="v3.2.4"
_BREEZE_ALLOWED_KIND_VERSIONS="v0.8.0"
_BREEZE_ALLOWED_MYSQL_VERSIONS="5.6 5.7"
diff --git a/scripts/ci/libraries/_initialization.sh
b/scripts/ci/libraries/_initialization.sh
index d3224e7..63041fd 100644
--- a/scripts/ci/libraries/_initialization.sh
+++ b/scripts/ci/libraries/_initialization.sh
@@ -205,7 +205,7 @@ function initialize_common_environment {
export VERSION_SUFFIX_FOR_SVN=""
# Default Kubernetes version
-export DEFAULT_KUBERNETES_VERSION="v1.18.2"
+export DEFAULT_KUBERNETES_VERSION="v1.18.6"
# Default KinD version
export DEFAULT_KIND_VERSION="v0.8.0"
diff --git a/scripts/ci/libraries/_kind.sh b/scripts/ci/libraries/_kind.sh
index 2cf43c3..173af6d 100644
--- a/scripts/ci/libraries/_kind.sh
+++ b/scripts/ci/libraries/_kind.sh
@@ -45,7 +45,7 @@ function make_sure_kubernetes_tools_are_installed() {
HELM_VERSION=${HELM_VERSION:=${DEFAULT_HELM_VERSION}}
HELM_URL="https://get.helm.sh/helm-${HELM_VERSION}-${SYSTEM}-amd64.tar.gz;
HELM_PATH="${BUILD_CACHE_DIR}/bin/helm"
-KUBECTL_VERSION=${KUBENETES_VERSION:=${DEFAULT_KUBERNETES_VERSION}}
+KUBECTL_VERSION=${KUBERNETES_VERSION:=${DEFAULT_KUBERNETES_VERSION}}
KUBECTL_URL="https://storage.googleapis.com/kubernetes-release/release/${KUBECTL_VERSION}/bin/${SYSTEM}/amd64/kubectl;
KUBECTL_PATH="${BUILD_CACHE_DIR}/bin/kubectl"
mkdir -pv "${BUILD_CACHE_DIR}/bin"
@@ -125,7 +125,7 @@ function delete_cluster() {
}
function perform_kind_cluster_operation() {
-ALLOWED_KIND_OPERATIONS="[ start restart stop deploy test shell ]"
+ALLOWED_KIND_OPERATIONS="[ start restart stop deploy test shell recreate ]"
set +u
if [[ -z "${1}" ]]; then
@@
[airflow] 16/32: Clean up tmp directory when exiting from breeze shell (#9930)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 213500eaebbad06cb6358d63330d7e82803e517b
Author: Jarek Potiuk
AuthorDate: Wed Jul 22 19:54:35 2020 +0200
Clean up tmp directory when exiting from breeze shell (#9930)
Since we are mountign tmp dir now to inside container, some
of the remnants of what's going on inside remains after exit.
This is particularly bad if you are using tmux (some of the
directories remaining there prevent tmux from re-run)
This change cleans up /tmp directory on exit from Breeze command.
It does it from inside container so that we clean up all
root-owned files without sudo.
(cherry picked from commit a9c871b47c017e97374efef64c02fdde9792aff5)
---
breeze | 1 +
scripts/ci/in_container/_in_container_utils.sh | 10 ++
scripts/ci/in_container/run_clear_tmp.sh| 21 +
.../tools/{ci_fix_ownership.sh => ci_clear_tmp.sh} | 5 ++---
scripts/ci/tools/ci_fix_ownership.sh| 3 ++-
5 files changed, 36 insertions(+), 4 deletions(-)
diff --git a/breeze b/breeze
index 27eaee0..91585a1 100755
--- a/breeze
+++ b/breeze
@@ -1963,6 +1963,7 @@ function run_breeze_command {
"${BUILD_CACHE_DIR}/${LAST_DC_PROD_FILE}" run --service-ports
--rm airflow "${@}"
else
"${BUILD_CACHE_DIR}/${LAST_DC_CI_FILE}" run --service-ports
--rm airflow "${@}"
+"${SCRIPTS_CI_DIR}/tools/ci_clear_tmp.sh"
fi
;;
run_exec)
diff --git a/scripts/ci/in_container/_in_container_utils.sh
b/scripts/ci/in_container/_in_container_utils.sh
index f2e94d4..946bd32 100644
--- a/scripts/ci/in_container/_in_container_utils.sh
+++ b/scripts/ci/in_container/_in_container_utils.sh
@@ -115,6 +115,16 @@ function in_container_fix_ownership() {
fi
}
+function in_container_clear_tmp() {
+if [[ ${VERBOSE} == "true" ]]; then
+echo "Cleaning ${AIRFLOW_SOURCES}/tmp from the container"
+fi
+rm -rf /tmp/*
+if [[ ${VERBOSE} == "true" ]]; then
+echo "Cleaned ${AIRFLOW_SOURCES}/tmp from the container"
+fi
+}
+
function in_container_go_to_airflow_sources() {
pushd "${AIRFLOW_SOURCES}" &>/dev/null || exit 1
}
diff --git a/scripts/ci/in_container/run_clear_tmp.sh
b/scripts/ci/in_container/run_clear_tmp.sh
new file mode 100755
index 000..324a795
--- /dev/null
+++ b/scripts/ci/in_container/run_clear_tmp.sh
@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# shellcheck source=scripts/ci/in_container/_in_container_script_init.sh
+. "$( dirname "${BASH_SOURCE[0]}" )/_in_container_script_init.sh"
+
+in_container_clear_tmp
diff --git a/scripts/ci/tools/ci_fix_ownership.sh
b/scripts/ci/tools/ci_clear_tmp.sh
similarity index 90%
copy from scripts/ci/tools/ci_fix_ownership.sh
copy to scripts/ci/tools/ci_clear_tmp.sh
index 8cde42d..9a8cb4a 100755
--- a/scripts/ci/tools/ci_fix_ownership.sh
+++ b/scripts/ci/tools/ci_clear_tmp.sh
@@ -36,10 +36,9 @@ HOST_OS="$(uname -s)"
export HOST_USER_ID
export HOST_GROUP_ID
export HOST_OS
-export BACKEND="sqlite"
docker-compose \
-f "${SCRIPTS_CI_DIR}/docker-compose/base.yml" \
-f "${SCRIPTS_CI_DIR}/docker-compose/local.yml" \
--f "${SCRIPTS_CI_DIR}/docker-compose/forward-credentials.yml" \
-run airflow /opt/airflow/scripts/ci/in_container/run_fix_ownership.sh
+ run --entrypoint /bin/bash \
+airflow -c /opt/airflow/scripts/ci/in_container/run_clear_tmp.sh
diff --git a/scripts/ci/tools/ci_fix_ownership.sh
b/scripts/ci/tools/ci_fix_ownership.sh
index 8cde42d..d3ae4ba 100755
--- a/scripts/ci/tools/ci_fix_ownership.sh
+++ b/scripts/ci/tools/ci_fix_ownership.sh
@@ -42,4 +42,5 @@ docker-compose \
-f "${SCRIPTS_CI_DIR}/docker-compose/base.yml" \
-f "${SCRIPTS_CI_DIR}/docker-compose/local.yml" \
-f "${SCRIPTS_CI_DIR}/docker-compose/forward-credentials.yml" \
-run airflow /opt/airflow/scripts/ci/in_container/run_fix_ownership.sh
+run --entrypoint /bin/bash \
+airflow -c
[airflow] 17/32: Pin google-cloud-container to <2 (#9901)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 6d10a7f71c6aa10f7020bd4b69355f36bc9179b7
Author: Ephraim Anierobi
AuthorDate: Mon Jul 20 22:56:37 2020 +0100
Pin google-cloud-container to <2 (#9901)
(cherry picked from commit 560e0b504b52ead405b604934893c784ddf4dafa)
---
Dockerfile.ci | 2 +-
setup.py | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/Dockerfile.ci b/Dockerfile.ci
index 5d4f240..2797e34 100644
--- a/Dockerfile.ci
+++ b/Dockerfile.ci
@@ -186,7 +186,7 @@ RUN mkdir -pv ${AIRFLOW_HOME} && \
mkdir -pv ${AIRFLOW_HOME}/logs
# Increase the value here to force reinstalling Apache Airflow pip dependencies
-ARG PIP_DEPENDENCIES_EPOCH_NUMBER="3"
+ARG PIP_DEPENDENCIES_EPOCH_NUMBER="4"
ENV PIP_DEPENDENCIES_EPOCH_NUMBER=${PIP_DEPENDENCIES_EPOCH_NUMBER}
# Optimizing installation of Cassandra driver
diff --git a/setup.py b/setup.py
index 1fe821b..e11ce70 100644
--- a/setup.py
+++ b/setup.py
@@ -255,7 +255,7 @@ gcp = [
'google-auth>=1.0.0, <2.0.0dev',
'google-auth-httplib2>=0.0.1',
'google-cloud-bigtable>=1.0.0',
-'google-cloud-container>=0.1.1',
+'google-cloud-container>=0.1.1,<2.0',
'google-cloud-dlp>=0.11.0',
'google-cloud-language>=1.1.1',
'google-cloud-secret-manager>=0.2.0',
[airflow] 27/32: Make XCom 2.7 compatible
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit ec1cb7da1464708ba83bbfe0bc5efc367ea361c3
Author: Daniel Imberman
AuthorDate: Mon Aug 3 14:23:18 2020 -0700
Make XCom 2.7 compatible
---
airflow/models/xcom.py | 8 +---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/airflow/models/xcom.py b/airflow/models/xcom.py
index 0b6a81d..9861c2c 100644
--- a/airflow/models/xcom.py
+++ b/airflow/models/xcom.py
@@ -234,7 +234,7 @@ class BaseXCom(Base, LoggingMixin):
raise
@staticmethod
-def deserialize_value(result) -> Any:
+def deserialize_value(result):
# TODO: "pickling" has been deprecated and JSON is preferred.
# "pickling" will be removed in Airflow 2.0.
enable_pickling = conf.getboolean('core', 'enable_xcom_pickling')
@@ -253,11 +253,13 @@ class BaseXCom(Base, LoggingMixin):
def resolve_xcom_backend():
"""Resolves custom XCom class"""
-clazz = conf.getimport("core", "xcom_backend",
fallback=f"airflow.models.xcom.{BaseXCom.__name__}")
+clazz = conf.getimport("core", "xcom_backend",
fallback="airflow.models.xcom.{}"
+ .format(BaseXCom.__name__))
if clazz:
if not issubclass(clazz, BaseXCom):
raise TypeError(
-f"Your custom XCom class `{clazz.__name__}` is not a subclass
of `{BaseXCom.__name__}`."
+"Your custom XCom class `{class_name}` is not a subclass of
`{base_name}`."
+.format(class_name=clazz.__name__, base_name=BaseXCom.__name__)
)
return clazz
return BaseXCom
[airflow] 23/32: Set pytest version to be < 6.0.0 due to breaking changes (#10043)
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 6f8b0ccd88a8f69b7f6efbc68adee15791daae72 Author: Felix Uellendall AuthorDate: Wed Jul 29 10:45:34 2020 +0200 Set pytest version to be < 6.0.0 due to breaking changes (#10043) The latest pytest version 6.0.0 released yesterday (2020-07-28) does not work in conjunction with the version of pylint (2.4.3) we are using. (cherry picked from commit 2e0d91d8eb9bcae1886358791917b953330a957f) --- setup.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.py b/setup.py index 35323d2..cc0f721 100644 --- a/setup.py +++ b/setup.py @@ -422,7 +422,7 @@ devel = [ 'paramiko', 'pre-commit', 'pysftp', -'pytest', +'pytest<6.0.0', # FIXME: pylint complaining for pytest.mark.* on v6.0 'pytest-cov', 'pytest-instafail', 'pytest-rerunfailures',
[airflow] 19/32: Fix bug in executor_config when defining resources (#9935)
This is an automated email from the ASF dual-hosted git repository.
kaxilnaik pushed a commit to branch v1-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 05ec21a22f84cdbe2aaed38b712c30f2cbb38b59
Author: Daniel Imberman
AuthorDate: Thu Jul 23 19:52:20 2020 -0700
Fix bug in executor_config when defining resources (#9935)
* Fix PodGenerator to handle Kubernetes resources
In Airflow 1.10.11, `namespaced['resources'] = resources` is missing.
This PR improves the definition of pod resources, `requests` and `limits`
are optional.
* Make it working in 2.7
* Add limit_gpu and fix ephemeral-storage keys
* Fix flake8
Co-authored-by: Riccardo Bini
---
airflow/kubernetes/pod.py | 4 +-
airflow/kubernetes/pod_generator.py| 33 ++---
tests/kubernetes/test_pod_generator.py | 132 +
3 files changed, 155 insertions(+), 14 deletions(-)
diff --git a/airflow/kubernetes/pod.py b/airflow/kubernetes/pod.py
index b1df462..0b332c2 100644
--- a/airflow/kubernetes/pod.py
+++ b/airflow/kubernetes/pod.py
@@ -33,7 +33,7 @@ class Resources(K8SModel):
:type request_memory: str
:param request_cpu: requested CPU number
:type request_cpu: float | str
-:param request_ephemeral_storage: requested ephermeral storage
+:param request_ephemeral_storage: requested ephemeral storage
:type request_ephemeral_storage: str
:param limit_memory: limit for memory usage
:type limit_memory: str
@@ -41,7 +41,7 @@ class Resources(K8SModel):
:type limit_cpu: float | str
:param limit_gpu: Limits for GPU used
:type limit_gpu: int
-:param limit_ephemeral_storage: Limit for ephermeral storage
+:param limit_ephemeral_storage: Limit for ephemeral storage
:type limit_ephemeral_storage: float | str
"""
def __init__(
diff --git a/airflow/kubernetes/pod_generator.py
b/airflow/kubernetes/pod_generator.py
index e46407b..d11c175 100644
--- a/airflow/kubernetes/pod_generator.py
+++ b/airflow/kubernetes/pod_generator.py
@@ -344,18 +344,26 @@ class PodGenerator(object):
resources = namespaced.get('resources')
if resources is None:
-requests = {
-'cpu': namespaced.get('request_cpu'),
-'memory': namespaced.get('request_memory'),
-'ephemeral-storage': namespaced.get('ephemeral-storage')
-}
-limits = {
-'cpu': namespaced.get('limit_cpu'),
-'memory': namespaced.get('limit_memory'),
-'ephemeral-storage': namespaced.get('ephemeral-storage')
-}
-all_resources = list(requests.values()) + list(limits.values())
-if all(r is None for r in all_resources):
+def extract(cpu, memory, ephemeral_storage, limit_gpu=None):
+resources_obj = {
+'cpu': namespaced.pop(cpu, None),
+'memory': namespaced.pop(memory, None),
+'ephemeral-storage': namespaced.pop(ephemeral_storage,
None),
+}
+if limit_gpu is not None:
+resources_obj['nvidia.com/gpu'] =
namespaced.pop(limit_gpu, None)
+
+resources_obj = {k: v for k, v in resources_obj.items() if v
is not None}
+
+if all(r is None for r in resources_obj):
+resources_obj = None
+return namespaced, resources_obj
+
+namespaced, requests = extract('request_cpu', 'request_memory',
'request_ephemeral_storage')
+namespaced, limits = extract('limit_cpu', 'limit_memory',
'limit_ephemeral_storage',
+ limit_gpu='limit_gpu')
+
+if requests is None and limits is None:
resources = None
else:
resources = k8s.V1ResourceRequirements(
@@ -371,6 +379,7 @@ class PodGenerator(object):
'iam.cloud.google.com/service-account': gcp_service_account_key
})
+namespaced['resources'] = resources
return PodGenerator(**namespaced).gen_pod()
@staticmethod
diff --git a/tests/kubernetes/test_pod_generator.py
b/tests/kubernetes/test_pod_generator.py
index 7d39cdc..d0faf4c 100644
--- a/tests/kubernetes/test_pod_generator.py
+++ b/tests/kubernetes/test_pod_generator.py
@@ -288,6 +288,138 @@ class TestPodGenerator(unittest.TestCase):
}, result)
@mock.patch('uuid.uuid4')
+def test_from_obj_with_resources(self, mock_uuid):
+self.maxDiff = None
+
+mock_uuid.return_value = self.static_uuid
+result = PodGenerator.from_obj({
+"KubernetesExecutor": {
+"annotations": {"test": "annotation"},
+"volumes": [
+{
+"name": "example-kubernetes-test-volume",
+
[airflow] 25/32: Fix docstrings in BigQueryGetDataOperator (#10042)
This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch v1-10-test in repository https://gitbox.apache.org/repos/asf/airflow.git commit 06b06d77e17ec29d74c59f0dae8da33698e286bb Author: Jinhui Zhang AuthorDate: Wed Jul 29 05:42:41 2020 -0700 Fix docstrings in BigQueryGetDataOperator (#10042) (cherry picked from commit 59cbff0874dd5318cda4b9ce7b7eeb1aad1dad4d) --- airflow/contrib/operators/bigquery_get_data.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/airflow/contrib/operators/bigquery_get_data.py b/airflow/contrib/operators/bigquery_get_data.py index f5e6e50..e16804b 100644 --- a/airflow/contrib/operators/bigquery_get_data.py +++ b/airflow/contrib/operators/bigquery_get_data.py @@ -56,7 +56,7 @@ class BigQueryGetDataOperator(BaseOperator): :type table_id: str :param max_results: The maximum number of records (rows) to be fetched from the table. (templated) -:type max_results: str +:type max_results: int :param selected_fields: List of fields to return (comma-separated). If unspecified, all fields are returned. :type selected_fields: str @@ -74,7 +74,7 @@ class BigQueryGetDataOperator(BaseOperator): def __init__(self, dataset_id, table_id, - max_results='100', + max_results=100, selected_fields=None, bigquery_conn_id='bigquery_default', delegate_to=None,
