potiuk commented on code in PR #35830:
URL: https://github.com/apache/airflow/pull/35830#discussion_r1405491778
##########
scripts/ci/pre_commit/common_precommit_utils.py:
##########
@@ -71,3 +80,74 @@ def get_directory_hash(directory: Path, skip_path_regexp:
str | None = None) ->
if file.is_file() and not file.name.startswith("."):
sha.update(file.read_bytes())
return sha.hexdigest()
+
+
+def initialize_breeze_precommit(name: str, file: str):
+ if name not in ("__main__", "__mp_main__"):
+ raise SystemExit(
+ "This file is intended to be executed as an executable program.
You cannot use it as a module."
+ f"To run this script, run the ./{file} command"
+ )
+
+ if os.environ.get("SKIP_BREEZE_PRE_COMMITS"):
+ console.print("[yellow]Skipping breeze pre-commit as
SKIP_BREEZE_PRE_COMMIT is set")
+ sys.exit(1)
+ if shutil.which("breeze") is None:
+ console.print(
+ "[red]The `breeze` command is not on path.[/]\n\n"
+ "[yellow]Please install breeze with `pipx install -e ./dev/breeze`
from Airflow sources "
+ "and make sure you run `pipx ensurepath`[/]\n\n"
+ "[bright_blue]You can also set SKIP_BREEZE_PRE_COMMITS env
variable to non-empty "
+ "value to skip all breeze tests."
+ )
+ sys.exit(1)
+
+
+def run_command_via_breeze_shell(
+ cmd: list[str],
+ python_version: str = DEFAULT_PYTHON_MAJOR_MINOR_VERSION,
+ backend: str = "none",
+ executor: str = "SequentialExecutor",
+ extra_env: dict[str, str] | None = None,
+ project_name: str = "pre-commit",
+ skip_environment_initialization: bool = True,
+ **other_popen_kwargs,
+) -> subprocess.CompletedProcess:
+ extra_env = extra_env or {}
+ subprocess_cmd: list[str] = [
+ "breeze",
+ "shell",
+ "--python",
+ python_version,
+ "--backend",
+ backend,
+ "--executor",
+ executor,
+ "--quiet",
+ "--skip-image-upgrade-check",
+ "--tty",
+ "disabled",
+ ]
+ if skip_environment_initialization:
+ subprocess_cmd.append("--skip-environment-initialization")
+ if project_name:
+ subprocess_cmd.extend(["--project-name", project_name])
+ subprocess_cmd.append(" ".join([shlex.quote(arg) for arg in cmd]))
+ if os.environ.get("VERBOSE_COMMANDS"):
+ console.print(
+ f"[magenta]Running command: {' '.join([shlex.quote(item) for item
in subprocess_cmd])}[/]"
+ )
+ return subprocess.run(
Review Comment:
1. This is deliberate choice and reflects the situation of many of our users
I want to avoid that. In many cases the image is out-dated and the user does
not want to pull/rebuild it because pulling/rebuilding requires a lot of
networking and tme. There are two cases for it:
a) when you are travelling/in-the-plane/no connectivity : I want to achieve
the goal that if you are travelling and you want to work "as usual" you should
not have to disable pre-commits when they are enabled.
b) when you have weak connectivity you want to rebuild your image with new
dependencies relatitvely rarely - this is always quite a bit network intensive
operation, even if it is heavily optimized by caching. Many of our contributors
(for example @ephraimbuddy - our release manager ) live in the place where
networking is much worse to what we are used to.
2. This is already added in a "soft" form - every time you run breeze
locally there is a check running to see if some of the code that "implies"
rebuilding of the image (Dockerfile, setup.* files, generated dependencies,
pyprojet.tomle) have been modified since the last time you built the image. And
you are gently asked to rebuild and warned that if you don't - weird things
might happen. And when you press `y` - the image will be rebuilt then. which
gives the user freedom on deciding when to rebuild the image. When they have
"weird error" - they will rebuild EVENTUALLY. And this is our goal - to let the
user an easy way to rebuild the image, but do not force them.
So the whole system is designed for eventual consistency, but also to leave
freedom to the user to decide if they prefer to do thing "fast" (or at all even
if they have no networking) with lass consistency or eventually "fully
consistently. It's been a deliberate decision and design documented in this
ADR:
https://github.com/apache/airflow/blob/main/dev/breeze/doc/adr/0012-asking-user-for-confirmation.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
