[GitHub] [airflow] uranusjr opened a new pull request, #28982: Resolve all variables in pickled XCom iterator

2023-01-16 Thread GitBox
uranusjr opened a new pull request, #28982: URL: https://github.com/apache/airflow/pull/28982 This is needed for cross-process communication. See discussion starting here: https://github.com/apache/airflow/pull/28191#issuecomment-1384931149 One thing I’m not sure about is the

[GitHub] [airflow] uranusjr commented on pull request #28191: Add custom pickling hooks to LazyXComAccess

2023-01-16 Thread GitBox
uranusjr commented on PR #28191: URL: https://github.com/apache/airflow/pull/28191#issuecomment-1384977262 Only when you want to pass results of a dynamic task into a subprocess (most commonly virtualenv, Docker, and Kubernetes, I think). -- This is an automated message from the Apache

[GitHub] [airflow] potiuk commented on pull request #28191: Add custom pickling hooks to LazyXComAccess

2023-01-16 Thread GitBox
potiuk commented on PR #28191: URL: https://github.com/apache/airflow/pull/28191#issuecomment-1384945654 > Edit: Found it. We need literal_binds. https://docs.sqlalchemy.org/en/14/faq/sqlexpressions.html#rendering-bound-parameters-inline Yeah. That's cool. Just wonder - is the

[GitHub] [airflow] uranusjr commented on pull request #28191: Add custom pickling hooks to LazyXComAccess

2023-01-16 Thread GitBox
uranusjr commented on PR #28191: URL: https://github.com/apache/airflow/pull/28191#issuecomment-1384938029 Yes it is. The SQLAlchemy says that should give us a fully compiled SQL, but apparently it does not actually. I don’t know why (or _how_ can we produce a fully compiled SQL). --

[GitHub] [airflow] potiuk commented on pull request #28191: Add custom pickling hooks to LazyXComAccess

2023-01-16 Thread GitBox
potiuk commented on PR #28191: URL: https://github.com/apache/airflow/pull/28191#issuecomment-1384934338 Isn't that SQL automatically generated by `query.statement.compile` from an ORM mapping ? At least that was my impression what happens here? -- This is an automated message from the

[GitHub] [airflow] uranusjr commented on pull request #28191: Add custom pickling hooks to LazyXComAccess

2023-01-16 Thread GitBox
uranusjr commented on PR #28191: URL: https://github.com/apache/airflow/pull/28191#issuecomment-1384931149 Actually I think this is something simpler and unrelated. With SQLite I get ``` INFO - sqlalchemy.exc.ProgrammingError: (sqlite3.ProgrammingError) Incorrect number of

[GitHub] [airflow] potiuk commented on pull request #28191: Add custom pickling hooks to LazyXComAccess

2023-01-16 Thread GitBox
potiuk commented on PR #28191: URL: https://github.com/apache/airflow/pull/28191#issuecomment-1384921406 cc: @pierrejeambrun @ephraimbuddy -> potential blocker for 2.5.1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [airflow] potiuk commented on pull request #28191: Add custom pickling hooks to LazyXComAccess

2023-01-16 Thread GitBox
potiuk commented on PR #28191: URL: https://github.com/apache/airflow/pull/28191#issuecomment-1384921032 cc: @uranusjr -> seems that MySQL hits us back again. I am not 100% sure whether this is a blocker to 2.5.1 - but it might be if it turns out that it is going to affect all MySQL users

[GitHub] [airflow] hterik commented on a diff in pull request #26639: Multi-threads support for processing diff queues in Kubernetes Executor

2023-01-16 Thread GitBox
hterik commented on code in PR #26639: URL: https://github.com/apache/airflow/pull/26639#discussion_r1071805534 ## airflow/executors/kubernetes_executor.py: ## @@ -634,13 +690,37 @@ def sync(self) -> None: self.log.debug("self.queued: %s", self.queued_tasks)

[GitHub] [airflow] hterik commented on a diff in pull request #26639: Multi-threads support for processing diff queues in Kubernetes Executor

2023-01-16 Thread GitBox
hterik commented on code in PR #26639: URL: https://github.com/apache/airflow/pull/26639#discussion_r1071796755 ## airflow/executors/kubernetes_executor.py: ## @@ -599,8 +655,31 @@ def sync(self) -> None: raise AirflowException(NOT_STARTED_MESSAGE)

[GitHub] [airflow] potiuk commented on a diff in pull request #28979: Fix rendering parameters in PapermillOperator

2023-01-16 Thread GitBox
potiuk commented on code in PR #28979: URL: https://github.com/apache/airflow/pull/28979#discussion_r1071795648 ## airflow/providers/papermill/operators/papermill.py: ## @@ -57,36 +59,43 @@ class PapermillOperator(BaseOperator): def __init__( self, *, -

[airflow] 01/01: Read logs from all containers in KPO

2023-01-16 Thread dstandish
This is an automated email from the ASF dual-hosted git repository. dstandish pushed a commit to branch mlnsharma/main in repository https://gitbox.apache.org/repos/asf/airflow.git commit 292c23283e3f97b8d281fbcd581c2f4d7abfa98e Author: Narasimha Sharma AuthorDate: Mon Jan 16 22:10:55 2023

[airflow] branch mlnsharma/main created (now 292c23283e)

2023-01-16 Thread dstandish
This is an automated email from the ASF dual-hosted git repository. dstandish pushed a change to branch mlnsharma/main in repository https://gitbox.apache.org/repos/asf/airflow.git at 292c23283e Read logs from all containers in KPO This branch includes the following new commits:

[GitHub] [airflow] mlnsharma opened a new pull request, #28981: Implemented log_containers option to read from all containers in KubernetesPodOperator

2023-01-16 Thread GitBox
mlnsharma opened a new pull request, #28981: URL: https://github.com/apache/airflow/pull/28981 Added a new 'log_containers' param to KubernetesPodOperator. This param can take a list of container names or a boolean True/False as input with default value being boolean False. If

[GitHub] [airflow] mlnsharma closed pull request #27802: Read logs from all containers in pod for KubernetesPodOperator

2023-01-16 Thread GitBox
mlnsharma closed pull request #27802: Read logs from all containers in pod for KubernetesPodOperator URL: https://github.com/apache/airflow/pull/27802 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [airflow] AndrewTsao commented on pull request #28191: Add custom pickling hooks to LazyXComAccess

2023-01-16 Thread GitBox
AndrewTsao commented on PR #28191: URL: https://github.com/apache/airflow/pull/28191#issuecomment-1384749135 Hi, I tested #28146 got another exception like this. My SQL version 5.7. Python 3.8, OS is centos 7.9. python modules list: ``` apache-airflow

[GitHub] [airflow] uranusjr commented on a diff in pull request #28979: Fix rendering parameters in PapermillOperator

2023-01-16 Thread GitBox
uranusjr commented on code in PR #28979: URL: https://github.com/apache/airflow/pull/28979#discussion_r1071672038 ## airflow/providers/papermill/operators/papermill.py: ## @@ -33,6 +33,8 @@ class NoteBook(File): """Jupyter notebook""" +template_fields: ClassVar =

[GitHub] [airflow] uranusjr opened a new pull request, #28980: Ensure non-JSON types go through encoder

2023-01-16 Thread GitBox
uranusjr opened a new pull request, #28980: URL: https://github.com/apache/airflow/pull/28980 Additional encoding logic is added for frozenset and set so the output is JSON-compatible. Also removed some unnecessary error handling. The default encoder simply raises TypeError, so we

[GitHub] [airflow] Taragolis commented on a diff in pull request #28979: Fix rendering parameters in PapermillOperator

2023-01-16 Thread GitBox
Taragolis commented on code in PR #28979: URL: https://github.com/apache/airflow/pull/28979#discussion_r1071666872 ## airflow/providers/papermill/operators/papermill.py: ## @@ -57,36 +59,43 @@ class PapermillOperator(BaseOperator): def __init__( self, *,

[GitHub] [airflow] Taragolis commented on pull request #28979: Fix rendering parameters in PapermillOperator

2023-01-16 Thread GitBox
Taragolis commented on PR #28979: URL: https://github.com/apache/airflow/pull/28979#issuecomment-1384729239 cc: @TPapajCin @marvinfretly @nicnguyen3103 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [airflow] Taragolis opened a new pull request, #28979: Fix rendering parameters in PapermillOperator

2023-01-16 Thread GitBox
Taragolis opened a new pull request, #28979: URL: https://github.com/apache/airflow/pull/28979 Add missing "parameters" into `NoteBook`'s templated fields and remove redundant loop on inlets/outlets related: #28977 -- This is an automated message from the Apache Git Service. To

[GitHub] [airflow] uranusjr opened a new pull request, #28978: Minor improvements to serde helpers

2023-01-16 Thread GitBox
uranusjr opened a new pull request, #28978: URL: https://github.com/apache/airflow/pull/28978 No functional changes, just some cleanups and docstring formatting. Extracted from #28742 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [airflow] uranusjr closed pull request #28742: Fix serialize() usages in custom JSON encoders

2023-01-16 Thread GitBox
uranusjr closed pull request #28742: Fix serialize() usages in custom JSON encoders URL: https://github.com/apache/airflow/pull/28742 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] uranusjr commented on pull request #28742: Fix serialize() usages in custom JSON encoders

2023-01-16 Thread GitBox
uranusjr commented on PR #28742: URL: https://github.com/apache/airflow/pull/28742#issuecomment-1384707382 Rethinking this, I _think_ this can simply be achieved by a better implemented `default`. I’ll split out that main change and other minor improvements here into two PRs so things are

[GitHub] [airflow] shyft-mike commented on a diff in pull request #28846: Updated app to support configuring the caching hash method for FIPS

2023-01-16 Thread GitBox
shyft-mike commented on code in PR #28846: URL: https://github.com/apache/airflow/pull/28846#discussion_r1071642334 ## airflow/www/app.py: ## @@ -131,7 +132,29 @@ def create_app(config=None, testing=False): init_robots(flask_app) +# Configure caching +

[GitHub] [airflow] github-actions[bot] commented on issue #28399: Dataproc Generator should allow creating spot instances

2023-01-16 Thread GitBox
github-actions[bot] commented on issue #28399: URL: https://github.com/apache/airflow/issues/28399#issuecomment-1384687968 This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further

[GitHub] [airflow] ferruzzi commented on a diff in pull request #28816: introduce a method to convert dictionaries to boto-style key-value lists

2023-01-16 Thread GitBox
ferruzzi commented on code in PR #28816: URL: https://github.com/apache/airflow/pull/28816#discussion_r1071626275 ## airflow/providers/amazon/aws/hooks/s3.py: ## @@ -42,6 +42,7 @@ from airflow.exceptions import AirflowException from airflow.providers.amazon.aws.exceptions

[GitHub] [airflow] potiuk closed issue #26931: Templating (like {{ ds }} ) stopped working in papermill after upgrade from 2.3.x to 2.4.x

2023-01-16 Thread GitBox
potiuk closed issue #26931: Templating (like {{ ds }} ) stopped working in papermill after upgrade from 2.3.x to 2.4.x URL: https://github.com/apache/airflow/issues/26931 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [airflow] Dinghang commented on a diff in pull request #26639: Multi-threads support for processing diff queues in Kubernetes Executor

2023-01-16 Thread GitBox
Dinghang commented on code in PR #26639: URL: https://github.com/apache/airflow/pull/26639#discussion_r1071622999 ## airflow/executors/kubernetes_executor.py: ## @@ -62,6 +64,50 @@ KubernetesWatchType = Tuple[str, str, Optional[str], Dict[str, str], str] +def

[GitHub] [airflow] potiuk commented on pull request #28548: Move comment to the right position

2023-01-16 Thread GitBox
potiuk commented on PR #28548: URL: https://github.com/apache/airflow/pull/28548#issuecomment-1384656059 Closing the change - seems not justified. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] potiuk closed pull request #28548: Move comment to the right position

2023-01-16 Thread GitBox
potiuk closed pull request #28548: Move comment to the right position URL: https://github.com/apache/airflow/pull/28548 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [airflow] hussein-awala commented on a diff in pull request #28961: Emit DataDog statsd metrics with metadata tags

2023-01-16 Thread GitBox
hussein-awala commented on code in PR #28961: URL: https://github.com/apache/airflow/pull/28961#discussion_r1071620930 ## airflow/config_templates/config.yml: ## @@ -851,6 +851,13 @@ metrics: type: string example: ~ default: "" +

[airflow] branch main updated (3adaa9b34e -> 65010fda09)

2023-01-16 Thread pierrejeambrun
This is an automated email from the ASF dual-hosted git repository. pierrejeambrun pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from 3adaa9b34e AWS system tests selective log purge (#28819) add 65010fda09 Fix for LocalKubernetesExecutor

[GitHub] [airflow] hussein-awala commented on a diff in pull request #28961: Emit DataDog statsd metrics with metadata tags

2023-01-16 Thread GitBox
hussein-awala commented on code in PR #28961: URL: https://github.com/apache/airflow/pull/28961#discussion_r1071620409 ## airflow/dag_processing/processor.py: ## @@ -484,7 +486,13 @@ def manage_slas(self, dag: DAG, session: Session = None) -> None:

[GitHub] [airflow] pierrejeambrun merged pull request #28638: Fix for LocalKubernetesExecutor scheduler is not serving logs

2023-01-16 Thread GitBox
pierrejeambrun merged PR #28638: URL: https://github.com/apache/airflow/pull/28638 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] Dinghang commented on a diff in pull request #26639: Multi-threads support for processing diff queues in Kubernetes Executor

2023-01-16 Thread GitBox
Dinghang commented on code in PR #26639: URL: https://github.com/apache/airflow/pull/26639#discussion_r1052930708 ## airflow/executors/kubernetes_executor.py: ## @@ -370,7 +416,17 @@ def sync(self) -> None: """ self.log.debug("Syncing KubernetesExecutor")

[GitHub] [airflow] ashb commented on a diff in pull request #28816: introduce a method to convert dictionaries to boto-style key-value lists

2023-01-16 Thread GitBox
ashb commented on code in PR #28816: URL: https://github.com/apache/airflow/pull/28816#discussion_r1071617889 ## airflow/providers/amazon/aws/hooks/s3.py: ## @@ -42,6 +42,7 @@ from airflow.exceptions import AirflowException from airflow.providers.amazon.aws.exceptions import

[GitHub] [airflow] Dinghang commented on pull request #26639: Multi-threads support for processing diff queues in Kubernetes Executor

2023-01-16 Thread GitBox
Dinghang commented on PR #26639: URL: https://github.com/apache/airflow/pull/26639#issuecomment-1384642241 Hi Team, gonna get some time addressing comments this week. Expect to reply back next week. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [airflow] snjypl commented on pull request #28638: Fix for LocalKubernetesExecutor scheduler is not serving logs

2023-01-16 Thread GitBox
snjypl commented on PR #28638: URL: https://github.com/apache/airflow/pull/28638#issuecomment-1384637725 > Could use a rebase before merging, it's 20+ commits behind master Done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [airflow] snjypl opened a new pull request, #28976: Migrate DagFileProcessorManager.clear_nonexistent_import_errors to internal API

2023-01-16 Thread GitBox
snjypl opened a new pull request, #28976: URL: https://github.com/apache/airflow/pull/28976 Fixes: #28785 --- **^ Add meaningful description above** Read the **[Pull Request

[GitHub] [airflow] potiuk closed pull request #28540: Using pypi proxy for speeding up development

2023-01-16 Thread GitBox
potiuk closed pull request #28540: Using pypi proxy for speeding up development URL: https://github.com/apache/airflow/pull/28540 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] potiuk commented on pull request #28540: Using pypi proxy for speeding up development

2023-01-16 Thread GitBox
potiuk commented on PR #28540: URL: https://github.com/apache/airflow/pull/28540#issuecomment-1384632819 Now when I think of it, I think this is not really needed, it seems like a really niche case. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [airflow] JGoldman110 commented on issue #28947: Status of testing of Apache Airflow 2.5.1rc1

2023-01-16 Thread GitBox
JGoldman110 commented on issue #28947: URL: https://github.com/apache/airflow/issues/28947#issuecomment-1384632106 #28788 - Swagger UI looks good -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] potiuk commented on pull request #28457: Add deferrable mode to DataprocCreateBatchOperator

2023-01-16 Thread GitBox
potiuk commented on PR #28457: URL: https://github.com/apache/airflow/pull/28457#issuecomment-1384630289 Needs conflict resolving. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] potiuk commented on pull request #28520: Set pip install opts in _PIP_ADDITIONAL_REQUIREMENTS

2023-01-16 Thread GitBox
potiuk commented on PR #28520: URL: https://github.com/apache/airflow/pull/28520#issuecomment-1384614065 closing this one as it seems to be achievable by PIP_* variables. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [airflow] potiuk closed pull request #28520: Set pip install opts in _PIP_ADDITIONAL_REQUIREMENTS

2023-01-16 Thread GitBox
potiuk closed pull request #28520: Set pip install opts in _PIP_ADDITIONAL_REQUIREMENTS URL: https://github.com/apache/airflow/pull/28520 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] vincbeck commented on a diff in pull request #28964: Add transfer operator S3 to (generic) SQL

2023-01-16 Thread GitBox
vincbeck commented on code in PR #28964: URL: https://github.com/apache/airflow/pull/28964#discussion_r1071602052 ## airflow/providers/amazon/aws/transfers/s3_to_sql.py: ## @@ -0,0 +1,158 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

[GitHub] [airflow] vincbeck commented on a diff in pull request #28964: Add transfer operator S3 to (generic) SQL

2023-01-16 Thread GitBox
vincbeck commented on code in PR #28964: URL: https://github.com/apache/airflow/pull/28964#discussion_r1071602052 ## airflow/providers/amazon/aws/transfers/s3_to_sql.py: ## @@ -0,0 +1,158 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

[GitHub] [airflow] potiuk commented on issue #27012: External python operator logging

2023-01-16 Thread GitBox
potiuk commented on issue #27012: URL: https://github.com/apache/airflow/issues/27012#issuecomment-1384609843 Are there any actions form that? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] potiuk commented on pull request #27802: Read logs from all containers in pod for KubernetesPodOperator

2023-01-16 Thread GitBox
potiuk commented on PR #27802: URL: https://github.com/apache/airflow/pull/27802#issuecomment-1384606835 CAn you solve the conflicts please @mlnsharma before others can take a look at it? -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [airflow] potiuk commented on issue #28452: TaskInstances do not succeed when using enable_logging=True option in DockerSwarmOperator

2023-01-16 Thread GitBox
potiuk commented on issue #28452: URL: https://github.com/apache/airflow/issues/28452#issuecomment-1384606025 Marked it as good first issue, hopefully someone wil take a look at this one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [airflow] harishkrao commented on a diff in pull request #28950: Sensor for Databricks partition and table changes

2023-01-16 Thread GitBox
harishkrao commented on code in PR #28950: URL: https://github.com/apache/airflow/pull/28950#discussion_r1071586211 ## airflow/providers/databricks/sensors/databricks.py: ## @@ -0,0 +1,231 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more

[airflow] branch main updated (cd637c223f -> 3adaa9b34e)

2023-01-16 Thread onikolas
This is an automated email from the ASF dual-hosted git repository. onikolas pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from cd637c223f Fix label name for `reauth` field in Docker Connection (#28974) add 3adaa9b34e AWS system tests

[GitHub] [airflow] o-nikolas merged pull request #28819: AWS system tests selective log purge

2023-01-16 Thread GitBox
o-nikolas merged PR #28819: URL: https://github.com/apache/airflow/pull/28819 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] pierrejeambrun commented on pull request #28638: Fix for LocalKubernetesExecutor scheduler is not serving logs

2023-01-16 Thread GitBox
pierrejeambrun commented on PR #28638: URL: https://github.com/apache/airflow/pull/28638#issuecomment-1384545009 Could use a rebase before merging, it's 20+ commits behind master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [airflow] ferruzzi commented on a diff in pull request #28816: introduce a method to convert dictionaries to boto-style key-value lists

2023-01-16 Thread GitBox
ferruzzi commented on code in PR #28816: URL: https://github.com/apache/airflow/pull/28816#discussion_r1071576124 ## airflow/providers/amazon/aws/hooks/s3.py: ## @@ -42,6 +42,7 @@ from airflow.exceptions import AirflowException from airflow.providers.amazon.aws.exceptions

[airflow] branch constraints-main updated: Updating constraints. Build id:

2023-01-16 Thread github-bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 66a94cc546 Updating

[GitHub] [airflow] ferruzzi commented on a diff in pull request #28816: introduce a method to convert dictionaries to boto-style key-value lists

2023-01-16 Thread GitBox
ferruzzi commented on code in PR #28816: URL: https://github.com/apache/airflow/pull/28816#discussion_r1071472110 ## airflow/providers/amazon/aws/hooks/s3.py: ## @@ -42,6 +42,7 @@ from airflow.exceptions import AirflowException from airflow.providers.amazon.aws.exceptions

[GitHub] [airflow] maggesssss commented on a diff in pull request #28964: Add transfer operator S3 to (generic) SQL

2023-01-16 Thread GitBox
magges commented on code in PR #28964: URL: https://github.com/apache/airflow/pull/28964#discussion_r1071573009 ## airflow/providers/amazon/aws/transfers/s3_to_sql.py: ## @@ -0,0 +1,158 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor

[GitHub] [airflow] ferruzzi commented on a diff in pull request #28819: AWS system tests selective log purge

2023-01-16 Thread GitBox
ferruzzi commented on code in PR #28819: URL: https://github.com/apache/airflow/pull/28819#discussion_r1071571584 ## tests/system/providers/amazon/aws/utils/__init__.py: ## @@ -249,7 +251,43 @@ def set_env_id() -> str: return env_id -def purge_logs( +def

[GitHub] [airflow] o-nikolas commented on pull request #28638: Fix for LocalKubernetesExecutor scheduler is not serving logs

2023-01-16 Thread GitBox
o-nikolas commented on PR #28638: URL: https://github.com/apache/airflow/pull/28638#issuecomment-1384527196 Anyone have some time to give this a second review/approval? Would be nice to get this merged for @snjypl Maybe @potiuk, @dstandish or @pierrejeambrun? -- This is an automated

[GitHub] [airflow] o-nikolas commented on pull request #28161: AIP-51 - Executor Coupling in Logging

2023-01-16 Thread GitBox
o-nikolas commented on PR #28161: URL: https://github.com/apache/airflow/pull/28161#issuecomment-1384526101 Anyone have some time to give this a second review/approval? Would be nice to get this merged for @snjypl Maybe @potiuk, @eladkal or @pierrejeambrun? -- This is an automated

[GitHub] [airflow] o-nikolas commented on a diff in pull request #28925: Add log for AWS Glue Job Console URL

2023-01-16 Thread GitBox
o-nikolas commented on code in PR #28925: URL: https://github.com/apache/airflow/pull/28925#discussion_r1071561782 ## airflow/providers/amazon/aws/operators/glue.py: ## @@ -144,6 +149,20 @@ def execute(self, context: Context): self.wait_for_completion, )

[GitHub] [airflow-site] potiuk commented on pull request #623: Remove Bash from the process of building the site

2023-01-16 Thread GitBox
potiuk commented on PR #623: URL: https://github.com/apache/airflow-site/pull/623#issuecomment-1384518413 There are a number of problems with dependencies the current approach has (and it caused us some problems in the recent past) so I think this is somethign we have to do anyway soon.

[GitHub] [airflow-site] potiuk commented on pull request #623: Remove Bash from the process of building the site

2023-01-16 Thread GitBox
potiuk commented on PR #623: URL: https://github.com/apache/airflow-site/pull/623#issuecomment-1384517156 H,,, I honestly think we should not invest too much in it. We are discussing about switching to Pelikan which is going to change the process completely anyway. -- This is an

[GitHub] [airflow] eladkal commented on issue #28938: Status of testing Providers that were prepared on January 14, 2023

2023-01-16 Thread GitBox
eladkal commented on issue #28938: URL: https://github.com/apache/airflow/issues/28938#issuecomment-1384512167 Docker provider will be excluded from this release and I will cut RC2 for it after this wave -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] vincbeck commented on a diff in pull request #28900: [WIP] Convert DagFileProcessor.execute_callbacks to Internal API

2023-01-16 Thread GitBox
vincbeck commented on code in PR #28900: URL: https://github.com/apache/airflow/pull/28900#discussion_r1071556627 ## airflow/api_internal/actions/dag.py: ## @@ -0,0 +1,42 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

[GitHub] [airflow] o-nikolas commented on a diff in pull request #28934: Remove hard-coded executor-database coupling

2023-01-16 Thread GitBox
o-nikolas commented on code in PR #28934: URL: https://github.com/apache/airflow/pull/28934#discussion_r1071551207 ## airflow/cli/commands/standalone_command.py: ## @@ -159,14 +159,28 @@ def calculate_env(self): # Make sure we're using a local executor flavour

[GitHub] [airflow] potiuk closed issue #28975: Additional info about Segmentation Violation in LocalTaskJob (#27381): @Taragolis

2023-01-16 Thread GitBox
potiuk closed issue #28975: Additional info about Segmentation Violation in LocalTaskJob (#27381): @Taragolis URL: https://github.com/apache/airflow/issues/28975 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [airflow] potiuk commented on issue #28975: Additional info about Segmentation Violation in LocalTaskJob (#27381): @Taragolis

2023-01-16 Thread GitBox
potiuk commented on issue #28975: URL: https://github.com/apache/airflow/issues/28975#issuecomment-1384495920 Wrong click I guess. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] potiuk commented on issue #28941: Tasks are in the queued status after restarting the redis container

2023-01-16 Thread GitBox
potiuk commented on issue #28941: URL: https://github.com/apache/airflow/issues/28941#issuecomment-1384493990 Then it nees to be looked up by somoene who knows celery and redis more than I do. I do not know redis that much - I guess there are delays in processing and saving the stored

[GitHub] [airflow] o-nikolas commented on a diff in pull request #28934: Remove hard-coded executor-database coupling

2023-01-16 Thread GitBox
o-nikolas commented on code in PR #28934: URL: https://github.com/apache/airflow/pull/28934#discussion_r1071544358 ## airflow/sensors/base.py: ## @@ -257,11 +258,14 @@ def _get_next_poke_interval( def prepare_for_execution(self) -> BaseOperator: task =

[airflow] branch main updated (dc3a3c7c52 -> cd637c223f)

2023-01-16 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from dc3a3c7c52 Add DataprocCancelOperationOperator (#28456) add cd637c223f Fix label name for `reauth` field in

[GitHub] [airflow] potiuk merged pull request #28974: Fix label name for `reauth` field in Docker Connection

2023-01-16 Thread GitBox
potiuk merged PR #28974: URL: https://github.com/apache/airflow/pull/28974 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[airflow] branch main updated (d24527bf75 -> dc3a3c7c52)

2023-01-16 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git from d24527bf75 Add documentation about cli `add connection` and AWS connection URI (#28852) add dc3a3c7c52 Add

[GitHub] [airflow] potiuk merged pull request #28456: Add DataprocCancelOperationOperator

2023-01-16 Thread GitBox
potiuk merged PR #28456: URL: https://github.com/apache/airflow/pull/28456 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] potiuk commented on issue #28947: Status of testing of Apache Airflow 2.5.1rc1

2023-01-16 Thread GitBox
potiuk commented on issue #28947: URL: https://github.com/apache/airflow/issues/28947#issuecomment-1384439268 All Good (I left one change for @webster-chainalys to confirm :) * https://github.com/apache/airflow/pull/27720 is difficult to trigger but this is just exception handling

[GitHub] [airflow] Taragolis commented on issue #28975: Additional info about Segmentation Violation in LocalTaskJob (#27381): @Taragolis

2023-01-16 Thread GitBox
Taragolis commented on issue #28975: URL: https://github.com/apache/airflow/issues/28975#issuecomment-1384396913 Oh.. It is probably miss click to "Convert to issue" on list of tasks. Isn't it? @potiuk -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] vandonr-amz commented on issue #28938: Status of testing Providers that were prepared on January 14, 2023

2023-01-16 Thread GitBox
vandonr-amz commented on issue #28938: URL: https://github.com/apache/airflow/issues/28938#issuecomment-1384396111 my Sagemaker changes were covered by a system test that's still green, so good to go :) -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] Taragolis commented on issue #28975: Additional info about Segmentation Violation in LocalTaskJob (#27381): @Taragolis

2023-01-16 Thread GitBox
Taragolis commented on issue #28975: URL: https://github.com/apache/airflow/issues/28975#issuecomment-1384394131  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [airflow] johannaojeling commented on pull request #28764: Add support for running a Beam Go pipeline with an executable binary

2023-01-16 Thread GitBox
johannaojeling commented on PR #28764: URL: https://github.com/apache/airflow/pull/28764#issuecomment-1384389915 @uranusjr can this be merged? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] Taragolis commented on a diff in pull request #28816: introduce a method to convert dictionaries to boto-style key-value lists

2023-01-16 Thread GitBox
Taragolis commented on code in PR #28816: URL: https://github.com/apache/airflow/pull/28816#discussion_r1071479962 ## airflow/providers/amazon/aws/hooks/s3.py: ## @@ -42,6 +42,7 @@ from airflow.exceptions import AirflowException from airflow.providers.amazon.aws.exceptions

[GitHub] [airflow] vandonr-amz commented on a diff in pull request #28816: introduce a method to convert dictionaries to boto-style key-value lists

2023-01-16 Thread GitBox
vandonr-amz commented on code in PR #28816: URL: https://github.com/apache/airflow/pull/28816#discussion_r1071478212 ## airflow/providers/amazon/aws/hooks/s3.py: ## @@ -42,6 +42,7 @@ from airflow.exceptions import AirflowException from airflow.providers.amazon.aws.exceptions

[GitHub] [airflow] ferruzzi commented on a diff in pull request #28816: introduce a method to convert dictionaries to boto-style key-value lists

2023-01-16 Thread GitBox
ferruzzi commented on code in PR #28816: URL: https://github.com/apache/airflow/pull/28816#discussion_r1071472110 ## airflow/providers/amazon/aws/hooks/s3.py: ## @@ -42,6 +42,7 @@ from airflow.exceptions import AirflowException from airflow.providers.amazon.aws.exceptions

[GitHub] [airflow] potiuk commented on issue #27328: SFTPOperator throws object of type 'PlainXComArg' has no len() when using with Taskflow API

2023-01-16 Thread GitBox
potiuk commented on issue #27328: URL: https://github.com/apache/airflow/issues/27328#issuecomment-1384371084 Woul you like to fix it @jtommi ? And @uranusjr - we both approved the change. I am starting to think that we should attempt to do do some automated prevention of similar

[GitHub] [airflow] potiuk commented on issue #27328: SFTPOperator throws object of type 'PlainXComArg' has no len() when using with Taskflow API

2023-01-16 Thread GitBox
potiuk commented on issue #27328: URL: https://github.com/apache/airflow/issues/27328#issuecomment-1384367659 Yep. It looks like this change https://github.com/apache/airflow/pull/2 broke it (the number indicates it's a bit hell-ish change). the checks are in constructor but should be

[GitHub] [airflow] Romsik788 commented on issue #28941: Tasks are in the queued status after restarting the redis container

2023-01-16 Thread GitBox
Romsik788 commented on issue #28941: URL: https://github.com/apache/airflow/issues/28941#issuecomment-1384367636 > But it's up to you to make it robust and production-ready and resilient to any kind of failures, I am afraid. Ok, I have deployed Apache Airflow on kubernetes 1.22 using the

[GitHub] [airflow] jtommi opened a new issue, #27328: SFTPOperator throws object of type 'PlainXComArg' has no len() when using with Taskflow API

2023-01-16 Thread GitBox
jtommi opened a new issue, #27328: URL: https://github.com/apache/airflow/issues/27328 ### Apache Airflow Provider(s) sftp ### Versions of Apache Airflow Providers apache-airflow-providers-sftp==4.1.0 ### Apache Airflow version 2.4.2 Python 3.10 ###

[airflow] branch main updated: Add documentation about cli `add connection` and AWS connection URI (#28852)

2023-01-16 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new d24527bf75 Add documentation about cli `add

[GitHub] [airflow] potiuk merged pull request #28852: Add documentation about cli `add connection` and AWS connection URI

2023-01-16 Thread GitBox
potiuk merged PR #28852: URL: https://github.com/apache/airflow/pull/28852 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #28852: Add documentation about cli `add connection` and AWS connection URI

2023-01-16 Thread GitBox
boring-cyborg[bot] commented on PR #28852: URL: https://github.com/apache/airflow/pull/28852#issuecomment-1384362745 Awesome work, congrats on your first merged pull request! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [airflow] rafalh commented on a diff in pull request #28970: Fix support for macros with dots in DataProcJobBuilder

2023-01-16 Thread GitBox
rafalh commented on code in PR #28970: URL: https://github.com/apache/airflow/pull/28970#discussion_r1071454703 ## tests/providers/google/cloud/hooks/test_dataproc.py: ## @@ -1006,17 +1006,9 @@ def test_set_python_main(self): self.builder.set_python_main(main)

[airflow] branch main updated: Mark license block in doc as text (#28965)

2023-01-16 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 4fff0555b5 Mark license block in doc as text

[GitHub] [airflow] potiuk merged pull request #28965: Mark license block in doc as text

2023-01-16 Thread GitBox
potiuk merged PR #28965: URL: https://github.com/apache/airflow/pull/28965 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] potiuk commented on pull request #28965: Mark license block in doc as text

2023-01-16 Thread GitBox
potiuk commented on PR #28965: URL: https://github.com/apache/airflow/pull/28965#issuecomment-1384337521 I wish there was `::code-block:: licence` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [airflow] potiuk merged pull request #28780: Fix hot loop with a burst of log messages (#28778)

2023-01-16 Thread GitBox
potiuk merged PR #28780: URL: https://github.com/apache/airflow/pull/28780 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] potiuk closed issue #28778: Script "clean-logs.sh" has an unexpected burst behavior

2023-01-16 Thread GitBox
potiuk closed issue #28778: Script "clean-logs.sh" has an unexpected burst behavior URL: https://github.com/apache/airflow/issues/28778 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #28780: Fix hot loop with a burst of log messages (#28778)

2023-01-16 Thread GitBox
boring-cyborg[bot] commented on PR #28780: URL: https://github.com/apache/airflow/pull/28780#issuecomment-1384335195 Awesome work, congrats on your first merged pull request! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[airflow] branch main updated: Fix hot loop with a burst of log messages (#28778) (#28780)

2023-01-16 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new 4b1a36f833 Fix hot loop with a burst of log

[GitHub] [airflow] csm10495 commented on pull request #28808: Allow setting the name for the base container within K8s Pod Operator

2023-01-16 Thread GitBox
csm10495 commented on PR #28808: URL: https://github.com/apache/airflow/pull/28808#issuecomment-1384319670 @eladkal can you check again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

  1   2   >