Busy week, couldn’t check it earlier… 3 comments:
(1) First general comment is the split in microsoft providers seems unnecessary to me: * * apache-airflow-backport-providers-microsoft-azure==2020.5.20rc2 * apache-airflow-backport-providers-microsoft-mssql==2020.5.20rc2 * apache-airflow-backport-providers-microsoft-winrm==2020.5.20rc2 Amazon and Google have just 1 single providers package. Is this intentional? (2) Second, the naming convention is now consistent which is great! Nit: airflow.providers.google.suite.operators.sheets.GoogleSheetsCreateSpreadsheet is an operator but doesn’t end with “Operator”. (3) Third, I wanted to validate at least importing all hooks/sensors/operators/etc. works correctly. Therefore I wrote a test script, which iterates over all providers and per backport package: 1. Runs a Python 3.7 Docker container 2. Does a fresh install of Airflow 1.10.10 3. Does a fresh install of the backport package 4. And tries importing each class (e.g. “from airflow.providers.amazon.aws.operators.ecs import ECSOperator”) The script is not 100% fool-proof and wildly inefficient, but should give us the bulk of the import errors. I did not test for cross-provider installations. Also uncertain if all import errors are related to the backports packages, or just fail in general, but we should at least check them out. The script is found here: https://gist.github.com/BasPH/31f192546d2728efc89b8ef9151e9652 And the import errors are found here: https://gist.github.com/BasPH/31f192546d2728efc89b8ef9151e9652#gistcomment-3315391 Bas On 23 May 2020, at 11:19, Driesprong, Fokko <fo...@driesprong.frl<mailto:fo...@driesprong.frl>> wrote: I have a few questions. I saw the thread and it would be great to move this forward. Currently I don't have time to test all the individual packages. I did take the celery one and ran the Release Audit Tool (RAT) of Apache: https://gist.github.com/Fokko/50bb812a4998e34f4274965276a60f8e Files with unapproved licenses: ./setup.cfg ./apache_airflow_backport_providers_celery.egg-info/SOURCES.txt ./apache_airflow_backport_providers_celery.egg-info/dependency_links.txt ./apache_airflow_backport_providers_celery.egg-info/not-zip-safe ./apache_airflow_backport_providers_celery.egg-info/requires.txt ./apache_airflow_backport_providers_celery.egg-info/top_level.txt ./licenses/LICENSE-d3js.txt ./licenses/LICENSE-flask-kerberos.txt Few remarks: - I think we should add a license to the setup.cfg - I don't know if we can add them to the egg-info, but I don't think that is necessary. What do others think? - Do we still use d3js? https://github.com/apache/airflow/search?q=d3js&unscoped_q=d3js Could be that this is still being pulled in recursively. - Same goes for flask-kerberos. The last two have a license, but they are not being recognized by the tool. - How did you generate the sha512? When I check the hash according to: https://www.apache.org/info/verification.html#CheckingHashes I get: shasum -a 512 apache-airflow-backport-providers-celery-2020.5.20rc2.tar.gz 46acd707b44ee40536e2342b3d59dbd14f3452158c9b300167b7f3e73e624e3439e76f298edd6ee2955e2569c3466db10d41141b1e0a0b73b9c5ad098fec43b9 apache-airflow-backport-providers-celery-2020.5.20rc2.tar.gz It looks different than the file: cat apache-airflow-backport-providers-celery-2020.5.20rc2-bin.tar.gz.sha512 dist/apache-airflow-backport-providers-celery-2020.5.20rc2-bin.tar.gz: 22E8D410 72BF03E2 7CE31489 8FF1FBD1 AE1D9AF9 1EEBC051 6EADC693 D216B58B 305EDB2B 27939B5A C983DF55 DDA33D50 FD4EFD85 3EA73AC6 4BDD9E42 EEC9A207 Could you elaborate on how to check the hashes, or better, have similar output as other projects: https://dist.apache.org/repos/dist/dev/avro/avro-1.9.2-rc2/avro-doc-1.9.2.tar.gz.sha512 Keys look good, but we should organise a key-signing party after COVID :) MacBook-Pro-van-Fokko:Downloads fokkodriesprong$ gpg --import KEYS.txt gpg: key 764129647BEC5C4B: "Chris Riccomini <criccom...@apache.org>" not changed gpg: key 35190B83D905A0BA: "Bolke de Bruin (CODE SIGNING KEY) < bo...@apache.org>" not changed gpg: key E6F0505CC7BC7E0D: "Maxime Beauchemin <maximebeauche...@apache.org>" not changed gpg: key 807C731A8C82A095: 1 signature not checked due to a missing key gpg: key 807C731A8C82A095: "Ash Berlin-Taylor <a...@apache.org>" not changed gpg: key DD7484A025F17494: "Kaxil Naik <kaxiln...@apache.org>" not changed gpg: key 75FCCD0A25FA0E4B: "Kaxil Naik <kaxiln...@gmail.com>" not changed gpg: key ADED48E098D58632: public key "Jarek Potiuk <ja...@potiuk.com>" imported gpg: Total number processed: 7 gpg: imported: 1 gpg: unchanged: 6 MacBook-Pro-van-Fokko:Downloads fokkodriesprong$ gpg --verify apache-airflow-backport-providers-celery-2020.5.20rc2-bin.tar.gz.asc apache-airflow-backport-providers-celery-2020.5.20rc2-bin.tar.gz gpg: Signature made di 19 mei 17:25:08 2020 CEST gpg: using RSA key 8622BEF5D4449830F6DA0111A8C74911009B9ED4 gpg: Good signature from "Jarek Potiuk <ja...@potiuk.com>" [unknown] gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 72A0 9BF8 53A4 49BB 1420 E631 ADED 48E0 98D5 8632 Subkey fingerprint: 8622 BEF5 D444 9830 F6DA 0111 A8C7 4911 009B 9ED4 Cheers, Fokko Op za 23 mei 2020 om 10:57 schreef Jarek Potiuk <jarek.pot...@polidea.com>: Whoa. One more PMC vote and we can finally release it .... On Sat, May 23, 2020 at 7:10 AM Kamil Breguła <kamil.breg...@polidea.com> wrote: +1 (non-binding) On Sat, May 23, 2020, 03:48 Kaxil Naik <kaxiln...@gmail.com> wrote: +1 (binding) On Fri, May 22, 2020 at 3:21 PM Jarek Potiuk <jarek.pot...@polidea.com wrote: Kind ping again :). The time is nearly up! On Thu, May 21, 2020 at 7:44 AM Jarek Potiuk < jarek.pot...@polidea.com wrote: Just a kind ping - Looking forward to some more testing and voting :). On Tue, May 19, 2020 at 6:15 PM Jarek Potiuk < jarek.pot...@polidea.com wrote: Hey all, I have cut Airflow Backport Providers 2020.5.20rc2. This email is calling a vote on the release, which will last for 72 hours - which means that it will end on Fri May 22 18:15:00 CEST 2020. The main differences from RC1 are (Thanks Ash for those comments and suggestions. it's the first time we do it for 58 packages so we better do it right from the beggining!): * descriptions are improved and cleaner and contain Python 3.6 * less repetitive package summary ("Back-ported airflow.providers.google.* package for Airflow 1.10.*) * for now cncf.kubernetes is removed (I also deleted rc1) until we make it works with 1.10 * wheel name contains only -py3 * additional info provided where we removed some classes from backports temporarily (google package with removed BigQuery) Consider this my (binding) +1. Airflow Backport Providers 2020.5.20rc2 are available at: https://dist.apache.org/repos/dist/dev/airflow/backport-providers/2020.5.20rc2/ *apache-airflow-backport-providers-2020.5.20rc2-source.tar.gz* is a source release that comes with INSTALL instructions. *apache-airflow-backport-providers-<PROVIDER>-2020.5.20rc2-bin.tar.gz* are the binary Python "sdist" release. Public keys are available at: https://dist.apache.org/repos/dist/release/airflow/KEYS Please vote accordingly: [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove with the reason Only votes from PMC members are binding, but members of the community are encouraged to test the release and vote with "(non-binding)". Please note that the version number excludes the 'rcX' string, so it's now simply 2020.5.20. This will allow us to rename the artifact without modifying the artifact checksums when we actually release. Each of the packages contains detailed changelog. Here is the list of links to the released packages and changelogs: https://pypi.org/project/apache-airflow-backport-providers-amazon/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-apache-cassandra/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-apache-druid/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-apache-hdfs/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-apache-hive/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-apache-livy/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-apache-pig/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-apache-pinot/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-apache-spark/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-apache-sqoop/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-celery/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-cloudant/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-databricks/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-datadog/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-dingding/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-discord/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-docker/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-elasticsearch/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-email/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-exasol/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-facebook/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-ftp/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-google/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-grpc/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-hashicorp/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-http/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-imap/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-jdbc/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-jenkins/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-jira/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-microsoft-azure/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-microsoft-mssql/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-microsoft-winrm/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-mongo/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-mysql/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-odbc/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-openfaas/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-opsgenie/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-oracle/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-pagerduty/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-papermill/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-postgres/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-presto/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-qubole/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-redis/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-salesforce/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-samba/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-segment/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-sftp/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-singularity/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-slack/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-snowflake/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-sqlite/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-ssh/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-vertica/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-yandex/2020.5.20rc2/ https://pypi.org/project/apache-airflow-backport-providers-zendesk/2020.5.20rc2/ Cheers, Jarek -- Jarek Potiuk Polidea | Principal Software Engineer M: +48 660 796 129 -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/> -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/> -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
