Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-06-12 Thread via GitHub
howardyoo commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2163029741 > I had a lot of conversation about open-telemetry and traces at Berlin Buzzwords and watched some talks and I really think it is going to make problem diagnosis and resolution much

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-06-11 Thread via GitHub
potiuk commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2161203256 I had a lot of conversation about open-telemetry and traces at Berlin Buzzwords and watched some talks and I really think it is going to make problem diagnosis and resolution much

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-06-11 Thread via GitHub
potiuk merged PR #37948: URL: https://github.com/apache/airflow/pull/37948 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-06-09 Thread via GitHub
potiuk commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2156502709 cc: @hussein-awala -> WDYT? I would love to merge that one now :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-06-08 Thread via GitHub
potiuk commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2156175784 Any comments @hussein-awala ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-06-02 Thread via GitHub
potiuk commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2143718995 > I'm not sure what we do usually means, here. Can you clarify? Do you mean we as Apache Airflow, or someone else (OTEL community) ? Also, I'm not sure what using two configs: class or

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-05-30 Thread via GitHub
howardyoo commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2139467681 > I'm very late to the party, but I have two comments: Hey @hussein-awala, thank you for your comments. It's pretty late alright, but I do welcome comments always! > > * why

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-05-29 Thread via GitHub
potiuk commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1619311763 ## newsfragments/37948.feature.rst: ## @@ -1,2 +1,2 @@ OpenTelemetry Traces for Apache Airflow -This new feature adds capability for Apache Airflow to emit 1)

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-05-29 Thread via GitHub
howardyoo commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2137452322 > One more request before we merge it @howardyoo -> could you please add newsfragment describing the traces? No problem! Just added and committed. -- This is an automated

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-05-29 Thread via GitHub
potiuk commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2137153918 One more request before we merge it @howardyoo -> could you please add newsfragment describing the traces? -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-05-28 Thread via GitHub
potiuk commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2134771031 Anyone wants to take another pass ? @ferruzzi @uranusjr ? Or should I just merge it ? -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-05-27 Thread via GitHub
howardyoo commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2134378466 Okay... ah.. On Tue, May 28, 2024 at 12:28 AM Jarek Potiuk ***@***.***> wrote: > Static test + docs fix left to go :) > > — > Reply to this email directly,

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-05-27 Thread via GitHub
potiuk commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2134371250 Static test + docs fix left to go :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-05-27 Thread via GitHub
howardyoo commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2134324736 > > Actually I reviewed the change and I would be ready to approve/merge that one when you rebase and fix all tests @howardyoo > > Thank you, @potiuk! I have fixed all

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-05-27 Thread via GitHub
howardyoo commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2134297561 > Actually I reviewed the change and I would be ready to approve/merge that one when you rebase and fix all tests @howardyoo Thank you, @potiuk! -- This is an automated

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-05-27 Thread via GitHub
potiuk commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2134005638 Actually I reviewed the change and I would be ready to approve/merge that one when you rebase and fix all tests @howardyoo -- This is an automated message from the Apache Git

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-04-14 Thread via GitHub
howardyoo commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2055263616 @potiuk , I am going to split this PR into part A, and the rest (Part B, C, D..), where part A would have the basic trace capabilities and necessary configuration, and part B will

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-04-14 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1565189532 ## airflow/dag_processing/manager.py: ## @@ -1029,6 +1051,25 @@ def _collect_results_from_processor(self, processor) -> None: )

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-04-14 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1565189095 ## airflow/traces/tracer.py: ## @@ -0,0 +1,280 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-04-14 Thread via GitHub
howardyoo closed pull request #37948: [AIP-49] OpenTelemetry Traces for Apache Airflow URL: https://github.com/apache/airflow/pull/37948 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-04-10 Thread via GitHub
uranusjr commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1560428293 ## airflow/dag_processing/manager.py: ## @@ -1029,6 +1051,25 @@ def _collect_results_from_processor(self, processor) -> None: )

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-04-10 Thread via GitHub
uranusjr commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1560427149 ## airflow/traces/tracer.py: ## @@ -0,0 +1,280 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-04-10 Thread via GitHub
dstandish commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2048324458 Yeah @howardyoo I think it's a good practice and will result in fewer errors. Big bang PRs, while sometimes necessary, have a tendency for things to fall between the cracks, bugs

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-04-10 Thread via GitHub
potiuk commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2047713345 That works, maybe even split it to smaller pieces. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-04-09 Thread via GitHub
howardyoo commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2046329683 > Also - purely practically - I think this change has a high-cherry-pick-breaking potential. We should deliberately start merging it from the changes that have the least potential

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-04-09 Thread via GitHub
howardyoo commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2046324781 > I tried to look at this PR finally and it is huge to review. I have a proposal though @howardyoo : Can you attempt to split split out adding traces separately for smaller subset a)

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-04-09 Thread via GitHub
potiuk commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2044682811 Also - purely practically - I think this change has a high-cherry-pick-breaking potential. We should deliberately start merging it from the changes that have the least potential

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-04-09 Thread via GitHub
potiuk commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2044624684 I tried to look at this PR finally and it is huge to review. I have a proposal though @howardyoo : Can you attempt to split split out adding traces separately for smaller subset a) add

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-04-09 Thread via GitHub
potiuk commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1557334927 ## airflow/config_templates/config.yml: ## @@ -1149,6 +1149,64 @@ metrics: type: string example: ~ default: "False" +traces: + description: | +

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-04-09 Thread via GitHub
potiuk commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1557333993 ## airflow/config_templates/config.yml: ## @@ -1149,6 +1149,64 @@ metrics: type: string example: ~ default: "False" +traces: + description: | +

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-31 Thread via GitHub
howardyoo closed pull request #37948: [AIP-49] OpenTelemetry Traces for Apache Airflow URL: https://github.com/apache/airflow/pull/37948 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-25 Thread via GitHub
ferruzzi commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-2019006751 Looks like you made a ton of progress last week. I'll try to make another (final??) pass in the next day or two. Thanks!! -- This is an automated message from the Apache Git

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-25 Thread via GitHub
ferruzzi commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1538293841 ## airflow/traces/otel_tracer.py: ## @@ -0,0 +1,316 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-25 Thread via GitHub
ferruzzi commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1538292247 ## airflow/traces/utils.py: ## @@ -0,0 +1,98 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-15 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1526937457 ## airflow/traces/otel_tracer.py: ## @@ -0,0 +1,316 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-15 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1526937064 ## airflow/traces/utils.py: ## @@ -0,0 +1,98 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-15 Thread via GitHub
ferruzzi commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1526570614 ## airflow/traces/utils.py: ## @@ -0,0 +1,98 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-15 Thread via GitHub
ferruzzi commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1526555009 ## airflow/traces/otel_tracer.py: ## @@ -0,0 +1,316 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-14 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1525761290 ## airflow/traces/utils.py: ## @@ -0,0 +1,98 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-14 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1525761290 ## airflow/traces/utils.py: ## @@ -0,0 +1,98 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-14 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1525754290 ## airflow/traces/utils.py: ## @@ -0,0 +1,98 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-14 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1525753637 ## airflow/traces/otel_tracer.py: ## @@ -0,0 +1,316 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-14 Thread via GitHub
ferruzzi commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1525342981 ## airflow/traces/tracer.py: ## @@ -0,0 +1,280 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-14 Thread via GitHub
ferruzzi commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1525319386 ## airflow/traces/otel_tracer.py: ## @@ -0,0 +1,316 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-14 Thread via GitHub
potiuk commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-1998090772 Actually `breeze static-checks --only-my-changes` should run WAY faster and do 9X% up to 100% of the job. -- This is an automated message from the Apache Git Service. To respond to

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-14 Thread via GitHub
ferruzzi commented on PR #37948: URL: https://github.com/apache/airflow/pull/37948#issuecomment-1997956996 We need to get these static checks passing and it looks like it's an issue that breeze should be able to autofix. When you get time, please run `breeze static-checks --all-files`.

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-14 Thread via GitHub
ferruzzi commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1525240559 ## airflow/traces/utils.py: ## @@ -18,52 +18,66 @@ from __future__ import annotations import logging +from typing import TYPE_CHECKING from

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-14 Thread via GitHub
ferruzzi commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1525237761 ## airflow/traces/otel_tracer.py: ## @@ -143,25 +135,24 @@ def start_span_from_dagrun( """Produce a span from dag run.""" # check if dagrun has

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-14 Thread via GitHub
ferruzzi commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1525236936 ## airflow/traces/otel_tracer.py: ## @@ -104,36 +98,34 @@ def start_span( links=None, start_time=None, ): -"""Start a span. if

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-14 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1524876807 ## airflow/jobs/scheduler_job_runner.py: ## @@ -1391,11 +1392,11 @@ def _start_queued_dagruns(self, session: Session) -> None: @span def

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-14 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1524874339 ## airflow/traces/otel_tracer.py: ## @@ -104,36 +98,34 @@ def start_span( links=None, start_time=None, ): -"""Start a span. if

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-14 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1524781847 ## airflow/traces/utils.py: ## @@ -18,52 +18,66 @@ from __future__ import annotations import logging +from typing import TYPE_CHECKING from

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-14 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1524768949 ## airflow/traces/tracer.py: ## @@ -0,0 +1,256 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-14 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1524758585 ## airflow/traces/tracer.py: ## @@ -0,0 +1,256 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-13 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1524186027 ## airflow/traces/utils.py: ## @@ -18,52 +18,66 @@ from __future__ import annotations import logging +from typing import TYPE_CHECKING from

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-13 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1524183403 ## airflow/traces/otel_tracer.py: ## @@ -104,36 +98,34 @@ def start_span( links=None, start_time=None, ): -"""Start a span. if

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-13 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1524176295 ## airflow/traces/otel_tracer.py: ## @@ -143,25 +135,24 @@ def start_span_from_dagrun( """Produce a span from dag run.""" # check if dagrun has

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-13 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1524169050 ## airflow/traces/utils.py: ## @@ -18,52 +18,66 @@ from __future__ import annotations import logging +from typing import TYPE_CHECKING from

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-13 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1524167075 ## airflow/jobs/job.py: ## @@ -211,35 +211,29 @@ def heartbeat( heartbeat_callback(session) self.log.debug("[heartbeat]")

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-13 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1524165158 ## airflow/jobs/job.py: ## @@ -211,35 +211,29 @@ def heartbeat( heartbeat_callback(session) self.log.debug("[heartbeat]")

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-13 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1524164314 ## airflow/executors/base_executor.py: ## @@ -311,12 +313,10 @@ def trigger_tasks(self, open_slots: int) -> None: @span def _process_tasks(self,

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-11 Thread via GitHub
ferruzzi commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1520538400 ## airflow/executors/base_executor.py: ## @@ -311,12 +313,10 @@ def trigger_tasks(self, open_slots: int) -> None: @span def _process_tasks(self,

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-11 Thread via GitHub
ferruzzi commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518281626 ## airflow/traces/otel_tracer.py: ## @@ -0,0 +1,333 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-11 Thread via GitHub
ferruzzi commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1520536156 ## airflow/traces/otel_tracer.py: ## @@ -0,0 +1,333 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-11 Thread via GitHub
ferruzzi commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518243419 ## airflow/jobs/scheduler_job_runner.py: ## @@ -1415,70 +1478,101 @@ def _schedule_dag_run( :param dag_run: The DagRun to schedule :return:

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-11 Thread via GitHub
ferruzzi commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1520533373 ## airflow/executors/base_executor.py: ## @@ -287,15 +301,41 @@ def trigger_tasks(self, open_slots: int) -> None: if key in self.attempts:

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-11 Thread via GitHub
ferruzzi commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1520529201 ## airflow/dag_processing/manager.py: ## @@ -1028,6 +1049,23 @@ def _collect_results_from_processor(self, processor) -> None: )

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-11 Thread via GitHub
ferruzzi commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1520526202 ## airflow/traces/tracer.py: ## @@ -0,0 +1,256 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-11 Thread via GitHub
ferruzzi commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1520517176 ## airflow/traces/utils.py: ## @@ -0,0 +1,105 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-11 Thread via GitHub
ferruzzi commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1520514584 ## scripts/ci/docker-compose/integration-otel.yml: ## @@ -54,14 +54,31 @@ services: - ./grafana/volume/dashboards:/grafana/dashboards -

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-11 Thread via GitHub
ferruzzi commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1520307917 ## airflow/jobs/job.py: ## @@ -211,35 +211,29 @@ def heartbeat( heartbeat_callback(session) self.log.debug("[heartbeat]")

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518483339 ## airflow/executors/base_executor.py: ## @@ -287,15 +301,41 @@ def trigger_tasks(self, open_slots: int) -> None: if key in self.attempts:

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518482023 ## airflow/executors/base_executor.py: ## @@ -287,15 +301,41 @@ def trigger_tasks(self, open_slots: int) -> None: if key in self.attempts:

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518481942 ## airflow/jobs/scheduler_job_runner.py: ## @@ -1415,70 +1478,101 @@ def _schedule_dag_run( :param dag_run: The DagRun to schedule :return:

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518480222 ## tests/core/test_otel_tracer.py: ## @@ -0,0 +1,155 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518480136 ## airflow/traces/tracer.py: ## @@ -0,0 +1,256 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518479352 ## airflow/traces/otel_tracer.py: ## @@ -0,0 +1,333 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518478981 ## airflow/traces/otel_tracer.py: ## @@ -0,0 +1,333 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518477892 ## airflow/traces/otel_tracer.py: ## @@ -0,0 +1,333 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518477669 ## airflow/traces/otel_tracer.py: ## @@ -0,0 +1,333 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518476652 ## airflow/traces/otel_tracer.py: ## @@ -0,0 +1,333 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518475273 ## airflow/jobs/scheduler_job_runner.py: ## @@ -1415,70 +1478,101 @@ def _schedule_dag_run( :param dag_run: The DagRun to schedule :return:

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518474943 ## airflow/executors/base_executor.py: ## @@ -287,15 +301,41 @@ def trigger_tasks(self, open_slots: int) -> None: if key in self.attempts:

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518474305 ## airflow/executors/base_executor.py: ## @@ -287,15 +301,41 @@ def trigger_tasks(self, open_slots: int) -> None: if key in self.attempts:

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518471941 ## airflow/dag_processing/manager.py: ## @@ -1028,6 +1049,23 @@ def _collect_results_from_processor(self, processor) -> None: )

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518471507 ## airflow/jobs/job.py: ## @@ -181,48 +182,67 @@ def heartbeat( :param session to use for saving the job """ previous_heartbeat =

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518471245 ## airflow/jobs/job.py: ## @@ -181,48 +182,67 @@ def heartbeat( :param session to use for saving the job """ previous_heartbeat =

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518463666 ## airflow/traces/tracer.py: ## @@ -0,0 +1,256 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518461942 ## airflow/traces/utils.py: ## @@ -0,0 +1,105 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518461640 ## airflow/traces/tracer.py: ## @@ -0,0 +1,256 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518461479 ## airflow/traces/utils.py: ## @@ -0,0 +1,105 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518461011 ## airflow/traces/utils.py: ## @@ -0,0 +1,105 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518459668 ## airflow/traces/utils.py: ## @@ -0,0 +1,105 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518458488 ## tests/core/test_otel_tracer.py: ## @@ -0,0 +1,155 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518444796 ## scripts/ci/docker-compose/integration-otel.yml: ## @@ -54,14 +54,31 @@ services: - ./grafana/volume/dashboards:/grafana/dashboards -

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518444257 ## airflow/traces/otel_tracer.py: ## @@ -0,0 +1,333 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518443375 ## airflow/traces/otel_tracer.py: ## @@ -0,0 +1,333 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518443166 ## airflow/config_templates/config.yml: ## @@ -1130,6 +1130,56 @@ metrics: type: string example: ~ default: "False" +traces: + description: |

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518442156 ## airflow/traces/otel_tracer.py: ## @@ -0,0 +1,333 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] [AIP-49] OpenTelemetry Traces for Apache Airflow [airflow]

2024-03-08 Thread via GitHub
howardyoo commented on code in PR #37948: URL: https://github.com/apache/airflow/pull/37948#discussion_r1518436591 ## airflow/executors/local_executor.py: ## @@ -261,6 +268,13 @@ def execute_async( if TYPE_CHECKING: assert

  1   2   >