potiuk commented on code in PR #36537:
URL: https://github.com/apache/airflow/pull/36537#discussion_r1443784202
##########
docs/apache-airflow/extra-packages-ref.rst:
##########
@@ -343,27 +350,278 @@ you need to list explicitly all the non-bundle extras
that you want to install.
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
| all_dbs | ``pip install 'apache-airflow[all_dbs]'`` |
All database integrations |
+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
-| devel | ``pip install 'apache-airflow[devel]'`` |
Minimum development dependencies (without Hadoop, Kerberos, providers) |
-+---------------------+-----------------------------------------------------+------------------------------------------------------------------------+
-| devel_hadoop | ``pip install 'apache-airflow[devel_hadoop]'`` |
Adds Hadoop stack libraries to ``devel`` dependencies |
+
+Development extras
+------------------
+
+Generally none of the ``devel`` extras install providers - they expect the
providers to be used from sources
+and those extras only make sense in editable mode. Users of Airflow should not
be using them, unless they
+start contributing back and install airflow from sources.
+
+Those extras are only available in Airflow when it is installed in editable
mode from sources
+(``pip install -e .``).
+
+Devel extras
+============
+
+The devel extras do not install dependencies for features of Airflow, but add
functionality that is needed to
+develop Airflow, such as running tests, static checks. They do not install
provider packages - even if they might be related
+to some providers (like ``devel_amazon``) but they might be needed if you want
to test code of thoe corresponding
+provider.
+
+Even if some ``devel`` extras relate to providers - they do not install
provider packages - for example
+``devel_amazon`` does not install amazon provider) but they might be needed if
you want to test code of
+the corresponding provider (for example running mypy checks or running tests).
+
++---------------------+-----------------------------------------+------------------------------------------------------+
+| extra | install command | enables
|
++=====================+=========================================+======================================================+
+| devel_amazon | pip install -e '.[devel_amazon]' | Adds all
test libraries needed to test amazon |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_azure | pip install -e '.[devel_azure]' | Adds all
test libraries needed to test azure |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_breeze | pip install -e '.[devel_breeze]' | Adds all
test libraries needed to test breeze |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_debuggers | pip install -e '.[devel_debuggers]' | Adds all
test libraries needed to test debuggers |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_deltalake | pip install -e '.[devel_deltalake]' | Adds all
test libraries needed to test deltalake |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_devscripts | pip install -e '.[devel_devscripts]' | Adds all
test libraries needed to test devscripts |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_duckdb | pip install -e '.[devel_duckdb]' | Adds all
test libraries needed to test duckdb |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_iceberg | pip install -e '.[devel_iceberg]' | Adds all
test libraries needed to test iceberg |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_mongo | pip install -e '.[devel_mongo]' | Adds all
test libraries needed to test mongo |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_mypy | pip install -e '.[devel_mypy]' | Adds all
test libraries needed to test mypy |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_sentry | pip install -e '.[devel_sentry]' | Adds all
test libraries needed to test sentry |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_static_checks | pip install -e '.[devel_static_checks]' | Adds all
test libraries needed to test static_checks |
++---------------------+-----------------------------------------+------------------------------------------------------+
+| devel_tests | pip install -e '.[devel_tests]' | Adds all
test libraries needed to test tests |
++---------------------+-----------------------------------------+------------------------------------------------------+
+
+Editable provider extras
+========================
+
+In order to test providers when installing Airflow in editable, development
mode, you need to install
+dependencies of the providers. This is done by installing the ``editable``
extra with ``pip install -e``.
+Those extras are not available in the released PyPI wheel packages, they are
only available when Airflow
+is installed locally in editable mode.
+
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| extra | install command
| enables |
++=================================+=====================================================+============================================================+
+| editable_airbyte | pip install -e '.[editable_airbyte]'
| Adds all libraries needed by the airbyte provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_alibaba | pip install -e '.[editable_alibaba]'
| Adds all libraries needed by the alibaba provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_amazon | pip install -e '.[editable_amazon]'
| Adds all libraries needed by the amazon provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_beam | pip install -e '.[editable_apache_beam]'
| Adds all libraries needed by the apache_beam provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_cassandra | pip install -e
'.[editable_apache_cassandra]' | Adds all libraries needed by the
apache_cassandra provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_drill | pip install -e '.[editable_apache_drill]'
| Adds all libraries needed by the apache_drill provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_druid | pip install -e '.[editable_apache_druid]'
| Adds all libraries needed by the apache_druid provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_flink | pip install -e '.[editable_apache_flink]'
| Adds all libraries needed by the apache_flink provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_hdfs | pip install -e '.[editable_apache_hdfs]'
| Adds all libraries needed by the apache_hdfs provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_hive | pip install -e '.[editable_apache_hive]'
| Adds all libraries needed by the apache_hive provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_impala | pip install -e '.[editable_apache_impala]'
| Adds all libraries needed by the apache_impala provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_kafka | pip install -e '.[editable_apache_kafka]'
| Adds all libraries needed by the apache_kafka provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_kylin | pip install -e '.[editable_apache_kylin]'
| Adds all libraries needed by the apache_kylin provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_livy | pip install -e '.[editable_apache_livy]'
| Adds all libraries needed by the apache_livy provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_pig | pip install -e '.[editable_apache_pig]'
| Adds all libraries needed by the apache_pig provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_pinot | pip install -e '.[editable_apache_pinot]'
| Adds all libraries needed by the apache_pinot provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apache_spark | pip install -e '.[editable_apache_spark]'
| Adds all libraries needed by the apache_spark provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_apprise | pip install -e '.[editable_apprise]'
| Adds all libraries needed by the apprise provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_arangodb | pip install -e '.[editable_arangodb]'
| Adds all libraries needed by the arangodb provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_asana | pip install -e '.[editable_asana]'
| Adds all libraries needed by the asana provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_atlassian_jira | pip install -e
'.[editable_atlassian_jira]' | Adds all libraries needed by the
atlassian_jira provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_celery | pip install -e '.[editable_celery]'
| Adds all libraries needed by the celery provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_cloudant | pip install -e '.[editable_cloudant]'
| Adds all libraries needed by the cloudant provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_cncf_kubernetes | pip install -e
'.[editable_cncf_kubernetes]' | Adds all libraries needed by the
cncf_kubernetes provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_cohere | pip install -e '.[editable_cohere]'
| Adds all libraries needed by the cohere provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_common_io | pip install -e '.[editable_common_io]'
| Adds all libraries needed by the common_io provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_common_sql | pip install -e '.[editable_common_sql]'
| Adds all libraries needed by the common_sql provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_databricks | pip install -e '.[editable_databricks]'
| Adds all libraries needed by the databricks provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_datadog | pip install -e '.[editable_datadog]'
| Adds all libraries needed by the datadog provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_dbt_cloud | pip install -e '.[editable_dbt_cloud]'
| Adds all libraries needed by the dbt_cloud provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_dingding | pip install -e '.[editable_dingding]'
| Adds all libraries needed by the dingding provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_discord | pip install -e '.[editable_discord]'
| Adds all libraries needed by the discord provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_docker | pip install -e '.[editable_docker]'
| Adds all libraries needed by the docker provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_elasticsearch | pip install -e '.[editable_elasticsearch]'
| Adds all libraries needed by the elasticsearch provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_exasol | pip install -e '.[editable_exasol]'
| Adds all libraries needed by the exasol provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_fab | pip install -e '.[editable_fab]'
| Adds all libraries needed by the fab provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_facebook | pip install -e '.[editable_facebook]'
| Adds all libraries needed by the facebook provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_ftp | pip install -e '.[editable_ftp]'
| Adds all libraries needed by the ftp provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_github | pip install -e '.[editable_github]'
| Adds all libraries needed by the github provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_google | pip install -e '.[editable_google]'
| Adds all libraries needed by the google provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_grpc | pip install -e '.[editable_grpc]'
| Adds all libraries needed by the grpc provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_hashicorp | pip install -e '.[editable_hashicorp]'
| Adds all libraries needed by the hashicorp provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_http | pip install -e '.[editable_http]'
| Adds all libraries needed by the http provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_imap | pip install -e '.[editable_imap]'
| Adds all libraries needed by the imap provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_influxdb | pip install -e '.[editable_influxdb]'
| Adds all libraries needed by the influxdb provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_jdbc | pip install -e '.[editable_jdbc]'
| Adds all libraries needed by the jdbc provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_jenkins | pip install -e '.[editable_jenkins]'
| Adds all libraries needed by the jenkins provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_microsoft_azure | pip install -e
'.[editable_microsoft_azure]' | Adds all libraries needed by the
microsoft_azure provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_microsoft_mssql | pip install -e
'.[editable_microsoft_mssql]' | Adds all libraries needed by the
microsoft_mssql provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_microsoft_psrp | pip install -e
'.[editable_microsoft_psrp]' | Adds all libraries needed by the
microsoft_psrp provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_microsoft_winrm | pip install -e
'.[editable_microsoft_winrm]' | Adds all libraries needed by the
microsoft_winrm provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_mongo | pip install -e '.[editable_mongo]'
| Adds all libraries needed by the mongo provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_mysql | pip install -e '.[editable_mysql]'
| Adds all libraries needed by the mysql provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_neo4j | pip install -e '.[editable_neo4j]'
| Adds all libraries needed by the neo4j provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_odbc | pip install -e '.[editable_odbc]'
| Adds all libraries needed by the odbc provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_openai | pip install -e '.[editable_openai]'
| Adds all libraries needed by the openai provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_openfaas | pip install -e '.[editable_openfaas]'
| Adds all libraries needed by the openfaas provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_openlineage | pip install -e '.[editable_openlineage]'
| Adds all libraries needed by the openlineage provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_opensearch | pip install -e '.[editable_opensearch]'
| Adds all libraries needed by the opensearch provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_opsgenie | pip install -e '.[editable_opsgenie]'
| Adds all libraries needed by the opsgenie provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_oracle | pip install -e '.[editable_oracle]'
| Adds all libraries needed by the oracle provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_pagerduty | pip install -e '.[editable_pagerduty]'
| Adds all libraries needed by the pagerduty provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_papermill | pip install -e '.[editable_papermill]'
| Adds all libraries needed by the papermill provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_pgvector | pip install -e '.[editable_pgvector]'
| Adds all libraries needed by the pgvector provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_pinecone | pip install -e '.[editable_pinecone]'
| Adds all libraries needed by the pinecone provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_postgres | pip install -e '.[editable_postgres]'
| Adds all libraries needed by the postgres provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_presto | pip install -e '.[editable_presto]'
| Adds all libraries needed by the presto provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_redis | pip install -e '.[editable_redis]'
| Adds all libraries needed by the redis provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_salesforce | pip install -e '.[editable_salesforce]'
| Adds all libraries needed by the salesforce provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_samba | pip install -e '.[editable_samba]'
| Adds all libraries needed by the samba provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_segment | pip install -e '.[editable_segment]'
| Adds all libraries needed by the segment provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_sendgrid | pip install -e '.[editable_sendgrid]'
| Adds all libraries needed by the sendgrid provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_sftp | pip install -e '.[editable_sftp]'
| Adds all libraries needed by the sftp provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_singularity | pip install -e '.[editable_singularity]'
| Adds all libraries needed by the singularity provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_slack | pip install -e '.[editable_slack]'
| Adds all libraries needed by the slack provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_smtp | pip install -e '.[editable_smtp]'
| Adds all libraries needed by the smtp provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_snowflake | pip install -e '.[editable_snowflake]'
| Adds all libraries needed by the snowflake provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_sqlite | pip install -e '.[editable_sqlite]'
| Adds all libraries needed by the sqlite provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_ssh | pip install -e '.[editable_ssh]'
| Adds all libraries needed by the ssh provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_tableau | pip install -e '.[editable_tableau]'
| Adds all libraries needed by the tableau provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_tabular | pip install -e '.[editable_tabular]'
| Adds all libraries needed by the tabular provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_telegram | pip install -e '.[editable_telegram]'
| Adds all libraries needed by the telegram provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_trino | pip install -e '.[editable_trino]'
| Adds all libraries needed by the trino provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_vertica | pip install -e '.[editable_vertica]'
| Adds all libraries needed by the vertica provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_weaviate | pip install -e '.[editable_weaviate]'
| Adds all libraries needed by the weaviate provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_yandex | pip install -e '.[editable_yandex]'
| Adds all libraries needed by the yandex provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+| editable_zendesk | pip install -e '.[editable_zendesk]'
| Adds all libraries needed by the zendesk provider |
++---------------------------------+-----------------------------------------------------+------------------------------------------------------------+
+
+Doc extras
Review Comment:
I moved `docs` down. Actually I have some plans to remove `doc` extra
eventually, I **think** It's not needed and holds us down a bit with CI image,
and I think we should move doc generation outside of our CI image - but I want
to do it as next step, this is quite a bit more changes to `breeze` and other
places, so I prefer to keep it separate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
