potiuk commented on a change in pull request #13767:
URL: https://github.com/apache/airflow/pull/13767#discussion_r561710341
##########
File path: dev/provider_packages/PROVIDER_INDEX_TEMPLATE.rst.jinja2
##########
@@ -0,0 +1,84 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+ NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE
+ OVERWRITTEN WHEN PREPARING PACKAGES.
+
+ IF YOU WANT TO MODIFY IT, YOU SHOULD MODIFY THE TEMPLATE
+ `PROVIDER_INDEX_TEMPLATE.rst.jinja2` IN the `provider_packages` DIRECTORY
+
+#}
+
+Package {{ PACKAGE_PIP_NAME }}
+------------------------------------------------------
+
+Release: {{ RELEASE }}{{ VERSION_SUFFIX }}
+
+Provider package
+----------------
+
+This is a provider package for ``{{PROVIDER_PACKAGE_ID}}`` provider. All
classes for this provider package
+are in ``{{FULL_PACKAGE_NAME}}`` python package.
+
+Installation
+------------
+
+.. note::
+
+ On November 2020, new version of PIP (20.3) has been released with a new,
2020 resolver. This resolver
+ does not yet work with Apache Airflow and might lead to errors in
installation - depends on your choice
+ of extras. In order to install Airflow you need to either downgrade pip to
version 20.2.4
+ ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you
need to add option
+ ``--use-deprecated legacy-resolver`` to your pip install command.
+
+
+You can install this package on top of an existing airflow 2.* installation via
+``pip install {{PACKAGE_PIP_NAME}}``
+{%- if PIP_REQUIREMENTS %}
+
+PIP requirements
+----------------
+
+{{ PIP_REQUIREMENTS_TABLE_RST | safe}}
+{%- endif %}
+{%- if CROSS_PROVIDERS_DEPENDENCIES %}
+
+Cross provider package dependencies
+-----------------------------------
+
+Those are dependencies that might be needed in order to use all the features
of the package.
+You need to install the specified backport providers package in order to use
them.
+
+You can install such cross-provider dependencies when installing from PyPI.
For example:
+
+.. code-block:: bash
+
+ pip install {{ PACKAGE_PIP_NAME }}[{{ CROSS_PROVIDERS_DEPENDENCIES[0] }}]
+
+
+{{ CROSS_PROVIDERS_DEPENDENCIES_TABLE_RST | safe }}
+
+{%- endif %}
Review comment:
> > feat!:, or fix!:, refactor!:, etc., which represent a breaking
change (indicated by the !) and will result in a SemVer major.
I think automatically generating nice changelog is something we can think
about in the future. And if you look at the commit changes now, there are many
we would have to exclude because they are just refactors/global updates. For
now I thin we should generate changelog manually (and every time someone makes
a new feature or breaking change - during review, we should ask the person to
add entry to the changelog. I will update the changelogs manually with the
first release, but from the 1.0.1 onwards, this is just a matter that all
committers will pay attention to it and make sure that Changelog is updated.
As the next step we could introduce the semantic commits and automated
changelog generation. But this would require discussion, voting and likely some
automation to make sure we follow it. I would like to decouple this from this
change.
##########
File path: airflow/providers/res.txt
##########
@@ -0,0 +1,61 @@
+./salesforce
Review comment:
Now. That's added by accident :)
##########
File path: airflow/providers/res.txt
##########
@@ -0,0 +1,61 @@
+./salesforce
Review comment:
No. That's added by accident :)
##########
File path: dev/provider_packages/PROVIDER_INDEX_TEMPLATE.rst.jinja2
##########
@@ -0,0 +1,84 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+ NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE
+ OVERWRITTEN WHEN PREPARING PACKAGES.
+
+ IF YOU WANT TO MODIFY IT, YOU SHOULD MODIFY THE TEMPLATE
+ `PROVIDER_INDEX_TEMPLATE.rst.jinja2` IN the `provider_packages` DIRECTORY
+
+#}
+
+Package {{ PACKAGE_PIP_NAME }}
+------------------------------------------------------
+
+Release: {{ RELEASE }}{{ VERSION_SUFFIX }}
+
+Provider package
+----------------
+
+This is a provider package for ``{{PROVIDER_PACKAGE_ID}}`` provider. All
classes for this provider package
+are in ``{{FULL_PACKAGE_NAME}}`` python package.
+
+Installation
+------------
+
+.. note::
+
+ On November 2020, new version of PIP (20.3) has been released with a new,
2020 resolver. This resolver
+ does not yet work with Apache Airflow and might lead to errors in
installation - depends on your choice
+ of extras. In order to install Airflow you need to either downgrade pip to
version 20.2.4
+ ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you
need to add option
+ ``--use-deprecated legacy-resolver`` to your pip install command.
+
+
+You can install this package on top of an existing airflow 2.* installation via
+``pip install {{PACKAGE_PIP_NAME}}``
+{%- if PIP_REQUIREMENTS %}
+
+PIP requirements
+----------------
+
+{{ PIP_REQUIREMENTS_TABLE_RST | safe}}
+{%- endif %}
+{%- if CROSS_PROVIDERS_DEPENDENCIES %}
+
+Cross provider package dependencies
+-----------------------------------
+
+Those are dependencies that might be needed in order to use all the features
of the package.
+You need to install the specified backport providers package in order to use
them.
+
+You can install such cross-provider dependencies when installing from PyPI.
For example:
+
+.. code-block:: bash
+
+ pip install {{ PACKAGE_PIP_NAME }}[{{ CROSS_PROVIDERS_DEPENDENCIES[0] }}]
+
+
+{{ CROSS_PROVIDERS_DEPENDENCIES_TABLE_RST | safe }}
+
+{%- endif %}
Review comment:
Just one comment here @kaxil @mik-laj about the naming/change process.
I am not totally against naming it UPDATING.md, but I think we are confusing
a bit airflow and providers release cycle. I think we should get the naming
right and content of this "manually written" parts.
I believe most of provider releases will be incremental updates - new
hooks/operators added basically or new (backwards-compatible) changes in the
operators. The changes that we have now in Google Providers are exception
rather than rule. We only need to add the "upgrading" guide because we make
backwards-incompatible changes due to 2.0 APIs. In vast majority of cases
people should be able to do 'pip install --upgrade
`apache-airflow-providers-<NN>` and there should be no need to follow any
"updating" guide. This is the main premise of semver, which communicates
breaking changes. The only time where we need to add "updating" information is
when we go 1.0.0 -> 2.0.0 etc.
Now - this means that vast majority of changes will be simply "new operator
"xxx" added". That's why I think CHANGELOG.rst is a good name, and this is also
a good name for the Heading in the docs.
While idally indeed we should be able to automaticaly get it from semantic
commits (and I am more and more convinced that we should switch to those), this
is a decision that will time to discuss and vote and apply and it will take
some time for people to adjust to, and we can do it as a follow-up. For now I
think this will be very little overhead if we just agree between committers,
that every time a new stuff is added to provider, we ask the PR author to add
entry in the CHANELOG.md and provider.yaml:
1) Add new version in provider.yaml patchelevel number (for bugfixes) or
minor version (for features)
2) Add a line in CHANGELOG.txt to describe the change added.
I think this makes it nicely distributed among the PR authors, and decreases
a lot the work needed by release manager (@kaxil :) - then PR authors and
commiters will take care about updating the versions and the role of release
manager will be just to review whether everything was included. Remember that
release manager will have to sometimes release 10-20 providers at a time and
mostly providers that the release manager knows very little about, so the less
work is put on the shoulders of that person, the better.
Let me know WDYT @kaxil @mik-laj and @ashb (I am mentioning @ashb here for
the release management part mainly).
##########
File path: BREEZE.rst
##########
@@ -2056,6 +2065,28 @@ This is the current syntax for `./breeze <./breeze>`_:
Flags:
+ -S, --version-suffix-for-pypi SUFFIX
+ Adds optional suffix to the version in the generated backport
package. It can be used
Review comment:
For regular providers too
##########
File path: BREEZE.rst
##########
@@ -2056,6 +2065,28 @@ This is the current syntax for `./breeze <./breeze>`_:
Flags:
+ -S, --version-suffix-for-pypi SUFFIX
+ Adds optional suffix to the version in the generated backport
package. It can be used
+ to generate rc1/rc2 ... versions of the packages to be uploaded to
PyPI.
+
+ -N, --version-suffix-for-svn SUFFIX
+ Adds optional suffix to the generated names of package. It can be
used to generate
+ rc1/rc2 ... versions of the packages to be uploaded to SVN.
+
+ --package-format PACKAGE_FORMAT
+
+ Chooses format of packages to prepare.
+
+ One of:
+
+ wheel,sdist,both
+
+ Default:
Review comment:
There is - wheel (though I might want to change it to both). I will see
why :)
##########
File path: airflow/providers/CHANGELOG.rst
##########
@@ -0,0 +1,25 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+Changelog
+---------
+
+1.0.0
+.....
+
+Initial version of the provider.
Review comment:
Hmmmmm . Strange indeed :)
##########
File path: airflow/providers/README.md
##########
@@ -1,28 +0,0 @@
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
- -->
-
-# Airflow Providers
Review comment:
Yes. All the READMEs are not needed any more - they are dynamically
generated while package is prepared and there is no point in keeping them in
repo.
##########
File path: airflow/providers/amazon/README.md
##########
@@ -1,483 +0,0 @@
-<!--
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
- -->
-
-
-# Package apache-airflow-providers-amazon
Review comment:
Yeah. Only during release. We do not really need them here. There is
very little value in keeping them in the repo, they will be available in PyPI.
But If you really think they are needed, I can add them back. I just thought -
since we already have the information there in setup.py, dependencies.json ,
the value of having it in the repo is very little. But I can be convince
otherwise..
##########
File path: airflow/providers/res.txt
##########
@@ -0,0 +1,61 @@
+./salesforce
+./celery
+./amazon
Review comment:
Debug file :)
##########
File path: dev/provider_packages/BACKPORT_PROVIDER_README_TEMPLATE.md.jinja2
##########
@@ -88,6 +95,6 @@ You can install such cross-provider dependencies when
installing from PyPI. For
pip install {{ PACKAGE_PIP_NAME }}[{{ CROSS_PROVIDERS_DEPENDENCIES[0] }}]
```
-{{ CROSS_PROVIDERS_DEPENDENCIES_TABLE }}
+{{ CROSS_PROVIDERS_DEPENDENCIES_TABLE | safe }}
Review comment:
Avoids escaping < > into > <
##########
File path: dev/provider_packages/BACKPORT_PROVIDER_README_TEMPLATE.md.jinja2
##########
@@ -88,6 +95,6 @@ You can install such cross-provider dependencies when
installing from PyPI. For
pip install {{ PACKAGE_PIP_NAME }}[{{ CROSS_PROVIDERS_DEPENDENCIES[0] }}]
```
-{{ CROSS_PROVIDERS_DEPENDENCIES_TABLE }}
+{{ CROSS_PROVIDERS_DEPENDENCIES_TABLE | safe }}
Review comment:
Avoids escaping < > into `>` `<`
##########
File path: dev/provider_packages/copy_provider_package_sources.py
##########
@@ -767,27 +781,38 @@ def do_refactor(self, in_process: bool = False) -> None:
# noqa
self.refactor_kubernetes_pod_operator()
# In order to debug Bowler - set in_process to True
self.qry.execute(write=True, silent=False, interactive=False,
in_process=in_process)
+ print("[green]OK[/]")
+
+
+def get_parser():
+ cli_parser = argparse.ArgumentParser(
+ description="Copies sources and optionally refactors provider code to
be Airflow 1.10 compatible.",
+ formatter_class=argparse.RawTextHelpFormatter,
+ )
+ cli_parser.add_argument(
+ "--backports",
+ action='store_true',
+ help=textwrap.dedent("Includes refactoring to prepare backport
packages rather than regular ones"),
+ )
+ cli_parser.add_argument(
+ "--debug",
+ action='store_true',
+ help=textwrap.dedent(
+ "Run bowler refactoring in single process. Makes it debuggable
with regular"
+ " IDE debugger (much slower)"
+ ),
+ )
+ return cli_parser
if __name__ == '__main__':
- BACKPORT_PACKAGES = os.getenv('BACKPORT_PACKAGES') == "true"
- in_process = False
- if len(sys.argv) > 1:
- if sys.argv[1] in ['--help', '-h']:
- print()
- print("Refactors provider packages to be Airflow 1.10 compatible.")
- print()
- print(f"Usage: {sys.argv[0]} [--debug] | [-h] | [--help]")
- print()
- print("You can use --debug flag in order to run bowler refactoring
in process.")
- print("This allows you to debug bowler process as usual using your
IDE debugger")
- print("Otherwise it heavily uses multi-processing and is
next-to-impossible to debug")
- print()
- print("Note - Bowler is also a lot slower in this mode.")
- print()
- sys.exit(0)
- if sys.argv[1] == '--debug':
- in_process = True
- copy_provider_sources()
- if BACKPORT_PACKAGES:
+ parser = get_parser()
+ args = parser.parse_args()
+
+ in_process = args.debug
+ copy_provider_sources(args.backports)
+ os.chdir(get_target_providers_folder())
+ if args.backports:
+ print("\nRefactoring code to be Airflow 1.10 - compatible\n")
RefactorBackportPackages().do_refactor(in_process=in_process)
+ print("\n[green]Refactored code successfully[/]\n")
Review comment:
:D
##########
File path: dev/provider_packages/prepare_provider_packages.py
##########
@@ -1225,184 +1195,272 @@ def
get_provider_info_from_provider_yaml(provider_package_id: str) -> Dict[str,
return provider_info
-def update_generated_files_for_package(
+def get_backport_current_changes_table(previous_release_commit_ref: str,
source_provider_package_path: str):
+ git_cmd = get_git_log_command(previous_release_commit_ref)
+ try:
+ changes = subprocess.check_output(git_cmd,
cwd=source_provider_package_path, universal_newlines=True)
+ changes_table = convert_git_changes_to_table(
+ previous_release_commit_ref, changes,
base_url="https://github.com/apache/airflow/commit/";
+ )
+ except subprocess.CalledProcessError:
+ changes_table = ''
+ return changes_table
+
+
+def get_version_tag(version: str, provider_package_id: str, version_suffix:
str):
+ return
f"providers-{provider_package_id.replace('.','-')}/{version}{version_suffix}"
+
+
+def print_changes_table(changes_table):
+ syntax = Syntax(changes_table, "rst", theme="light")
+ console = Console(width=200)
+ console.print(syntax)
+
+
+def get_all_changes_for_regular_packages(
+ versions: List[str],
provider_package_id: str,
- current_release_version: str,
+ source_provider_package_path: str,
version_suffix: str,
- imported_classes: List[str],
- backport_packages: bool,
- update_release_notes: bool,
- update_setup: bool,
-) -> Tuple[int, int]:
- """
- Updates release notes (BACKPORT_PROVIDER_README.md/README.md) for the
package.
- Returns Tuple of total number of entities and badly named entities.
+) -> Tuple[bool, str]:
+ current_version = versions[0]
+ current_tag = get_version_tag(current_version, provider_package_id,
version_suffix)
+ print(f"Checking if tag `{current_tag}` exists.")
+ if not subprocess.call(
+ get_git_tag_check_command(current_tag),
+ cwd=source_provider_package_path,
+ stderr=subprocess.DEVNULL,
+ ):
+ print(f"The tag {current_tag} exists.")
+ # The tag already exists
+ changes = subprocess.check_output(
+ get_git_log_command(current_tag, 'HEAD'),
+ cwd=source_provider_package_path,
+ universal_newlines=True,
+ )
+ if len(changes) > 0:
+ print(
+ f"[yellow]The provider {provider_package_id} has changes"
+ f" since last release but version is not updated[/]"
+ )
+ print()
+ print(
+ f'Please update version in
`airflow/providers/{provider_package_id.replace("-","/")}/'
+ 'provider.yaml` to prepare release.\n'
+ )
+ changes_table = convert_git_changes_to_table(
+ "UNKNOWN", changes,
base_url="https://github.com/apache/airflow/commit/";, markdown=False
+ )
+ print_changes_table(changes_table)
+ return False, changes_table
+ else:
+ print(f"No changes for {provider_package_id}")
+ return False, ""
+ print("The tag does not exist. ")
+ if len(versions) == 1:
+ print("This provider has never been released!\n")
+ else:
+ print(f"The {provider_package_id} is ready to be released!\n")
+ current_version = 'HEAD'
+ changes_table = ''
+ print_version = versions[0]
+ for version in versions[1:]:
+ version_tag = get_version_tag(version, provider_package_id,
version_suffix)
+ changes = subprocess.check_output(
+ get_git_log_command(version_tag, current_version),
+ cwd=source_provider_package_path,
+ universal_newlines=True,
+ )
+ changes_table += convert_git_changes_to_table(
+ print_version, changes,
base_url="https://github.com/apache/airflow/commit/";, markdown=False
+ )
+ current_version = version
+ print_version = version
+ print_changes_table(changes_table)
+ return True, changes_table
+
+
+def get_provider_details(provider_package_id: str) -> ProviderPackageDetails:
+ return ProviderPackageDetails(
+ provider_package_id=provider_package_id,
+ full_package_name=f"airflow.providers.{provider_package_id}",
+
source_provider_package_path=get_source_package_path(provider_package_id),
+
documentation_provider_package_path=get_documentation_package_path(provider_package_id),
+ )
- :param provider_package_id: id of the package
- :param current_release_version: release version:
- :param version_suffix: version suffix corresponding to the version in the
code
- :param imported_classes - entities that have been imported from providers
- :param backport_packages: whether to prepare regular (False) or backport
(True) packages
- :param update_release_notes: whether to update release notes
- :param update_setup: whether to update setup files
- :return: Tuple of total/bad number of entities
- """
- verify_provider_package(provider_package_id)
- full_package_name = f"airflow.providers.{provider_package_id}"
- source_provider_package_path = get_source_package_path(provider_package_id)
- entity_summaries = get_package_class_summary(full_package_name,
imported_classes)
- past_releases = get_all_releases(
- provider_package_path=source_provider_package_path,
backport_packages=backport_packages
+def get_provider_jinja_context(
+ provider_details: ProviderPackageDetails,
+ current_release_version: str,
+ version_suffix: str,
+ backport_packages: bool,
+):
+ verify_provider_package(provider_details.provider_package_id)
+ cross_providers_dependencies = get_cross_provider_dependent_packages(
+ provider_package_id=provider_details.provider_package_id
)
- current_release_version, previous_release = check_if_release_version_ok(
- past_releases, current_release_version, backport_packages
+ release_version_no_leading_zeros =
strip_leading_zeros(current_release_version)
+ pip_requirements_table = convert_pip_requirements_to_table(
+ PROVIDERS_REQUIREMENTS[provider_details.provider_package_id]
)
- cross_providers_dependencies = get_cross_provider_dependent_packages(
- provider_package_id=provider_package_id
+ pip_requirements_table_rst = convert_pip_requirements_to_table(
+ PROVIDERS_REQUIREMENTS[provider_details.provider_package_id],
markdown=False
)
- previous_release = get_previous_release_info(
- previous_release_version=previous_release,
- past_releases=past_releases,
- current_release_version=current_release_version,
+ cross_providers_dependencies_table =
convert_cross_package_dependencies_to_table(
+ cross_providers_dependencies,
+ backport_packages=backport_packages,
)
- release_version_no_leading_zeros = (
- strip_leading_zeros_in_calver(current_release_version)
- if backport_packages
- else current_release_version
+ cross_providers_dependencies_table_rst =
convert_cross_package_dependencies_to_table(
+ cross_providers_dependencies, backport_packages=backport_packages,
markdown=False
)
context: Dict[str, Any] = {
"ENTITY_TYPES": list(EntityType),
"README_FILE": "BACKPORT_PROVIDER_README.md" if backport_packages else
"README.md",
- "PROVIDER_PACKAGE_ID": provider_package_id,
- "PACKAGE_PIP_NAME": get_pip_package_name(provider_package_id,
backport_packages),
- "FULL_PACKAGE_NAME": full_package_name,
- "PROVIDER_PATH": full_package_name.replace(".", "/"),
+ "PROVIDER_PACKAGE_ID": provider_details.provider_package_id,
+ "PACKAGE_PIP_NAME": get_pip_package_name(
+ provider_details.provider_package_id,
backport_packages=backport_packages
+ ),
+ "FULL_PACKAGE_NAME": provider_details.full_package_name,
+ "PROVIDER_PATH": provider_details.full_package_name.replace(".", "/"),
"RELEASE": current_release_version,
"RELEASE_NO_LEADING_ZEROS": release_version_no_leading_zeros,
"VERSION_SUFFIX": version_suffix,
- "ADDITIONAL_INFO":
get_additional_package_info(provider_package_path=source_provider_package_path),
+ "ADDITIONAL_INFO": get_additional_package_info(
+ provider_package_path=provider_details.source_provider_package_path
+ ),
+ "CHANGELOG": get_changelog_for_package(
+ provider_package_path=provider_details.source_provider_package_path
+ ),
"CROSS_PROVIDERS_DEPENDENCIES": cross_providers_dependencies,
- "PIP_REQUIREMENTS": PROVIDERS_REQUIREMENTS[provider_package_id],
- "PROVIDER_TYPE": "Backport provider" if BACKPORT_PACKAGES else
"Provider",
- "PROVIDERS_FOLDER": "backport-providers" if BACKPORT_PACKAGES else
"providers",
+ "PIP_REQUIREMENTS":
PROVIDERS_REQUIREMENTS[provider_details.provider_package_id],
+ "PROVIDER_TYPE": "Backport provider" if backport_packages else
"Provider",
+ "PROVIDERS_FOLDER": "backport-providers" if backport_packages else
"providers",
"INSTALL_REQUIREMENTS": get_install_requirements(
- provider_package_id=provider_package_id,
backport_packages=backport_packages
+ provider_package_id=provider_details.provider_package_id,
backport_packages=backport_packages
),
"SETUP_REQUIREMENTS": get_setup_requirements(),
"EXTRAS_REQUIREMENTS": get_package_extras(
- provider_package_id=provider_package_id,
backport_packages=backport_packages
+ provider_package_id=provider_details.provider_package_id,
backport_packages=backport_packages
),
- "PROVIDER_INFO":
get_provider_info_from_provider_yaml(provider_package_id),
+ "CROSS_PROVIDERS_DEPENDENCIES_TABLE":
cross_providers_dependencies_table,
+ "CROSS_PROVIDERS_DEPENDENCIES_TABLE_RST":
cross_providers_dependencies_table_rst,
+ "PIP_REQUIREMENTS_TABLE": pip_requirements_table,
+ "PIP_REQUIREMENTS_TABLE_RST": pip_requirements_table_rst,
}
- if update_release_notes:
- git_cmd = get_git_command(previous_release)
- try:
- changes = subprocess.check_output(
- git_cmd, cwd=source_provider_package_path,
universal_newlines=True
- )
- changes_table = convert_git_changes_to_table(
- changes, base_url="https://github.com/apache/airflow/commit/";
- )
- except subprocess.CalledProcessError:
- # TODO(potiuk) fix me for both backport/provider package check
- changes_table = ''
- context["CURRENT_CHANGES_TABLE"] = changes_table
- pip_requirements_table = convert_pip_requirements_to_table(
- PROVIDERS_REQUIREMENTS[provider_package_id]
- )
- cross_providers_dependencies_table =
convert_cross_package_dependencies_to_table(
- cross_providers_dependencies,
- backport_packages=backport_packages,
- )
- context["CROSS_PROVIDERS_DEPENDENCIES_TABLE"] =
cross_providers_dependencies_table
- context["PIP_REQUIREMENTS_TABLE"] = pip_requirements_table
+ return context
- total, bad = check_if_classes_are_properly_named(entity_summaries)
+
+def prepare_readme_file(
+ context,
+):
+ readme_content = LICENCE
+ readme_template_name = PROVIDER_TEMPLATE_PREFIX + "README"
+ readme_content += render_template(template_name=readme_template_name,
context=context, extension='.md')
+ readme_file_path = os.path.join(TARGET_PROVIDER_PACKAGES_PATH, "README.md")
+ with open(readme_file_path, "wt") as readme_file:
+ readme_file.write(readme_content)
+
+
+def update_generated_files_for_regular_package(
+ provider_package_id: str,
+ version_suffix: str,
+ update_release_notes: bool,
+ update_setup: bool,
+) -> None:
+ """
+ Updates generated files (readme, changes and/or
setup.cfg/setup.py/manifest.in/provider_info)
+
+ :param provider_package_id: id of the package
+ :param version_suffix: version suffix corresponding to the version in the
code
+ :param update_release_notes: whether to update release notes
+ :param update_setup: whether to update setup files
+ """
+ verify_provider_package(provider_package_id)
+ provider_details = get_provider_details(provider_package_id)
+ provider_info = get_provider_info_from_provider_yaml(provider_package_id)
+ versions: List[str] = provider_info['versions']
+ current_release_version = versions[0]
+ # previous_release = versions[1] if len(versions) > 1 else None
+ # previous_release_commit_ref =
f"providers-{provider_package_id.replace('.','-')}/{previous_release}"
Review comment:
Yep
##########
File path: BREEZE.rst
##########
@@ -2056,6 +2065,28 @@ This is the current syntax for `./breeze <./breeze>`_:
Flags:
+ -S, --version-suffix-for-pypi SUFFIX
+ Adds optional suffix to the version in the generated backport
package. It can be used
+ to generate rc1/rc2 ... versions of the packages to be uploaded to
PyPI.
+
+ -N, --version-suffix-for-svn SUFFIX
+ Adds optional suffix to the generated names of package. It can be
used to generate
+ rc1/rc2 ... versions of the packages to be uploaded to SVN.
+
+ --package-format PACKAGE_FORMAT
+
+ Chooses format of packages to prepare.
+
+ One of:
+
+ wheel,sdist,both
+
+ Default:
Review comment:
Changed to both. The problem was with extra 's' in variable name
##########
File path: dev/provider_packages/PROVIDER_INDEX_TEMPLATE.rst.jinja2
##########
@@ -0,0 +1,84 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+ NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE
+ OVERWRITTEN WHEN PREPARING PACKAGES.
+
+ IF YOU WANT TO MODIFY IT, YOU SHOULD MODIFY THE TEMPLATE
+ `PROVIDER_INDEX_TEMPLATE.rst.jinja2` IN the `provider_packages` DIRECTORY
+
+#}
+
+Package {{ PACKAGE_PIP_NAME }}
+------------------------------------------------------
+
+Release: {{ RELEASE }}{{ VERSION_SUFFIX }}
+
+Provider package
+----------------
+
+This is a provider package for ``{{PROVIDER_PACKAGE_ID}}`` provider. All
classes for this provider package
+are in ``{{FULL_PACKAGE_NAME}}`` python package.
+
+Installation
+------------
+
+.. note::
+
+ On November 2020, new version of PIP (20.3) has been released with a new,
2020 resolver. This resolver
+ does not yet work with Apache Airflow and might lead to errors in
installation - depends on your choice
+ of extras. In order to install Airflow you need to either downgrade pip to
version 20.2.4
+ ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you
need to add option
+ ``--use-deprecated legacy-resolver`` to your pip install command.
+
+
+You can install this package on top of an existing airflow 2.* installation via
+``pip install {{PACKAGE_PIP_NAME}}``
+{%- if PIP_REQUIREMENTS %}
+
+PIP requirements
+----------------
+
+{{ PIP_REQUIREMENTS_TABLE_RST | safe}}
+{%- endif %}
+{%- if CROSS_PROVIDERS_DEPENDENCIES %}
+
+Cross provider package dependencies
+-----------------------------------
+
+Those are dependencies that might be needed in order to use all the features
of the package.
+You need to install the specified backport providers package in order to use
them.
+
+You can install such cross-provider dependencies when installing from PyPI.
For example:
+
+.. code-block:: bash
+
+ pip install {{ PACKAGE_PIP_NAME }}[{{ CROSS_PROVIDERS_DEPENDENCIES[0] }}]
+
+
+{{ CROSS_PROVIDERS_DEPENDENCIES_TABLE_RST | safe }}
+
+{%- endif %}
Review comment:
Also one small comment here.
Looking at
https://apache-airflow.slack.com/archives/CCR6P6JRL/p1611225089006700 - this is
why I think we should have the detailed commit log (as a separate document
linked from the index). Users are sometimes aware of the PRs they are
interested in and want to know in which version they were included. And it's
not very easy to find it out. We will have a separate set of tags for each
provider, so for some people, it would be great to have it in this nicely
formatted document rather than having to run some complex git commands to find
out.
##########
File path: dev/provider_packages/PROVIDER_INDEX_TEMPLATE.rst.jinja2
##########
@@ -0,0 +1,84 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+ NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE
+ OVERWRITTEN WHEN PREPARING PACKAGES.
+
+ IF YOU WANT TO MODIFY IT, YOU SHOULD MODIFY THE TEMPLATE
+ `PROVIDER_INDEX_TEMPLATE.rst.jinja2` IN the `provider_packages` DIRECTORY
+
+#}
+
+Package {{ PACKAGE_PIP_NAME }}
+------------------------------------------------------
+
+Release: {{ RELEASE }}{{ VERSION_SUFFIX }}
+
+Provider package
+----------------
+
+This is a provider package for ``{{PROVIDER_PACKAGE_ID}}`` provider. All
classes for this provider package
+are in ``{{FULL_PACKAGE_NAME}}`` python package.
+
+Installation
+------------
+
+.. note::
+
+ On November 2020, new version of PIP (20.3) has been released with a new,
2020 resolver. This resolver
+ does not yet work with Apache Airflow and might lead to errors in
installation - depends on your choice
+ of extras. In order to install Airflow you need to either downgrade pip to
version 20.2.4
+ ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you
need to add option
+ ``--use-deprecated legacy-resolver`` to your pip install command.
+
+
+You can install this package on top of an existing airflow 2.* installation via
+``pip install {{PACKAGE_PIP_NAME}}``
+{%- if PIP_REQUIREMENTS %}
+
+PIP requirements
+----------------
+
+{{ PIP_REQUIREMENTS_TABLE_RST | safe}}
+{%- endif %}
+{%- if CROSS_PROVIDERS_DEPENDENCIES %}
+
+Cross provider package dependencies
+-----------------------------------
+
+Those are dependencies that might be needed in order to use all the features
of the package.
+You need to install the specified backport providers package in order to use
them.
+
+You can install such cross-provider dependencies when installing from PyPI.
For example:
+
+.. code-block:: bash
+
+ pip install {{ PACKAGE_PIP_NAME }}[{{ CROSS_PROVIDERS_DEPENDENCIES[0] }}]
+
+
+{{ CROSS_PROVIDERS_DEPENDENCIES_TABLE_RST | safe }}
+
+{%- endif %}
Review comment:
I will make it separate page then next to index.rst and link to it.
##########
File path: dev/provider_packages/PROVIDER_INDEX_TEMPLATE.rst.jinja2
##########
@@ -0,0 +1,84 @@
+{#
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+ NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE
+ OVERWRITTEN WHEN PREPARING PACKAGES.
+
+ IF YOU WANT TO MODIFY IT, YOU SHOULD MODIFY THE TEMPLATE
+ `PROVIDER_INDEX_TEMPLATE.rst.jinja2` IN the `provider_packages` DIRECTORY
+
+#}
+
+Package {{ PACKAGE_PIP_NAME }}
+------------------------------------------------------
+
+Release: {{ RELEASE }}{{ VERSION_SUFFIX }}
+
+Provider package
+----------------
+
+This is a provider package for ``{{PROVIDER_PACKAGE_ID}}`` provider. All
classes for this provider package
+are in ``{{FULL_PACKAGE_NAME}}`` python package.
+
+Installation
+------------
+
+.. note::
+
+ On November 2020, new version of PIP (20.3) has been released with a new,
2020 resolver. This resolver
+ does not yet work with Apache Airflow and might lead to errors in
installation - depends on your choice
+ of extras. In order to install Airflow you need to either downgrade pip to
version 20.2.4
+ ``pip install --upgrade pip==20.2.4`` or, in case you use Pip 20.3, you
need to add option
+ ``--use-deprecated legacy-resolver`` to your pip install command.
+
+
+You can install this package on top of an existing airflow 2.* installation via
+``pip install {{PACKAGE_PIP_NAME}}``
+{%- if PIP_REQUIREMENTS %}
+
+PIP requirements
+----------------
+
+{{ PIP_REQUIREMENTS_TABLE_RST | safe}}
+{%- endif %}
+{%- if CROSS_PROVIDERS_DEPENDENCIES %}
+
+Cross provider package dependencies
+-----------------------------------
+
+Those are dependencies that might be needed in order to use all the features
of the package.
+You need to install the specified backport providers package in order to use
them.
+
+You can install such cross-provider dependencies when installing from PyPI.
For example:
+
+.. code-block:: bash
+
+ pip install {{ PACKAGE_PIP_NAME }}[{{ CROSS_PROVIDERS_DEPENDENCIES[0] }}]
+
+
+{{ CROSS_PROVIDERS_DEPENDENCIES_TABLE_RST | safe }}
+
+{%- endif %}
Review comment:
I will also add a few hyperlinks here and there, so that we will be able
to freely jump between those different pages -
* README in PyPI,
* index.rst in docs
* commit changes in docs
Those will be interlinked with version numbers so that we will be able to
jump from PYPI 2.0.0 package straight to detailed commit log for 2.0.0 and
2.0.0 Changelog (and all the cross-links between those).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
