potiuk commented on code in PR #36537: URL: https://github.com/apache/airflow/pull/36537#discussion_r1443788324
########## hatch_build.py: ########## @@ -0,0 +1,144 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +from __future__ import annotations + +import logging +import os +from pathlib import Path +from subprocess import run +from typing import Any, Callable, Iterable + +from hatchling.builders.config import BuilderConfig +from hatchling.builders.hooks.plugin.interface import BuildHookInterface +from hatchling.builders.plugin.interface import BuilderInterface +from hatchling.plugin.manager import PluginManager + +log = logging.getLogger(__name__) +log_level = logging.getLevelName(os.getenv("CUSTOM_AIRFLOW_BUILD_LOG_LEVEL", "INFO")) +log.setLevel(log_level) + + +class CustomBuild(BuilderInterface[BuilderConfig, PluginManager]): + """Custom build class for Airflow assets and git version.""" + + PLUGIN_NAME = "custom" Review Comment: Nope. It's not possible. This is (current) limitation of hatch. The plugin will be named `custom` anyway if we define it in `build_hatch.py`. We can use different name if we build and release our own `airflow-build` package containing the extension and release it in PyPI - then piugins can have any names. But if you use it in `build_hatch.py` it will anyhow be `custom`: https://hatch.pypa.io/latest/plugins/build-hook/custom/#example <img width="913" alt="Screenshot 2024-01-06 at 16 26 50" src="https://github.com/apache/airflow/assets/595491/7d90b33f-9fee-42f2-8662-d555d093b21f";> If we build more custom plugins in the future - we **might** want to decide to release the package (I even already built such package locally and used it, so I know how to do it) - but it has the disadvantage, that you have to manually build the package and add special configuration in your pyproject.toml to use it from that locally build .whl file, which means that iteration on it is a bit slow. Using `build_hatch.py` has the advantage that you can change the code and it will be used automatically in the next `pip install` or `hatch build` without rebuilding the separate .whl file with our plugins. So once the code settles and we keep it running for some time, we might do it, but it also has some other serious disadvantages, so I'd rather keep the `hatch_build.py` for a long time until we release at least few MINOR versions of airlfow (and maybe forever). The problem with releasing build plugins for build backend, is that when we release thepackage so that anyone will be able to use it. It will also make it more difficult to fix anything and if you release it as a package you need to version it and make it works for old and new versions of airflow (because then, it becomes `build dependency`). Which basically means that we need to explicitly limit such dependency to `< MAJOR` because otherwise we might break installation of past airflow versions when we introduce breaking changes in new versions. You have to be really, really, really careful with such `build plugins` because you can accidentally break past installation of your software when you release a new version of your build plugin. And those problems can be easily overlooked see for example https://github.com/apache/airflow/issues/32672 where new Cython 4 release broke ALL airflow installations with `pymssql` extra on ARM - precisely because the build dependency of `pymssql` did not have MAJOR limitation on CPython and it was not compatible with it. So after Cython 4 was released, pip while building pymssql package (it did not have .whl packages) downloaded CPython 4 and tried to use it to build pymssql and failed. Those are very subtle errors - so I stongly prefer custom build code to be part of the "airflow sources" package, not externally versioned build package. This could happen if somoene would use `sdist` package rather than .whl to build already released version of Airflow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
