[GitHub] [airflow] xinbinhuang edited a comment on pull request #17987: AwsBaseHook make client_type optional in get_client_type

2021-09-02 Thread GitBox
xinbinhuang edited a comment on pull request #17987: URL: https://github.com/apache/airflow/pull/17987#issuecomment-912276682 `.get_resource_type` has the same problem. Would be nice to change it here as well. It will be even better! if we can refactor them somehow, it seems really

[GitHub] [airflow] github-actions[bot] commented on pull request #17987: AwsBaseHook make client_type optional in get_client_type

2021-09-02 Thread GitBox
github-actions[bot] commented on pull request #17987: URL: https://github.com/apache/airflow/pull/17987#issuecomment-912277055 The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not

[GitHub] [airflow] xinbinhuang edited a comment on pull request #17987: AwsBaseHook make client_type optional in get_client_type

2021-09-02 Thread GitBox
xinbinhuang edited a comment on pull request #17987: URL: https://github.com/apache/airflow/pull/17987#issuecomment-912276682 `.get_resource_type` has the same problem. Would be nice to change it here as well. It will be even better! if we can refactor them somehow, it seems really

[GitHub] [airflow] xinbinhuang commented on pull request #17987: AwsBaseHook make client_type optional in get_client_type

2021-09-02 Thread GitBox
xinbinhuang commented on pull request #17987: URL: https://github.com/apache/airflow/pull/17987#issuecomment-912276682 `.get_resource_type` has the same problem. Would be nice to change it here as well. It will be even better if we can refactor them somehow, it seems really

[GitHub] [airflow] xinbinhuang opened a new pull request #14640: Allow ExternalTaskSensor to wait for taskgroup

2021-09-02 Thread GitBox
xinbinhuang opened a new pull request #14640: URL: https://github.com/apache/airflow/pull/14640 closes: #14563 This PR enables ExternalTaskSensor to also wait for the external task_group. The implementation is to retrieve the external DAG from the DagBag and then

[GitHub] [airflow] uranusjr commented on a change in pull request #17989: Require timetable class be registered via plugin

2021-09-02 Thread GitBox
uranusjr commented on a change in pull request #17989: URL: https://github.com/apache/airflow/pull/17989#discussion_r701573209 ## File path: airflow/serialization/serialized_objects.py ## @@ -128,22 +122,39 @@ def decode_timezone(var: Union[str, int]) -> Timezone: return

[airflow] branch main updated (ca4f99d -> 1e1b3de)

2021-09-02 Thread eladkal
This is an automated email from the ASF dual-hosted git repository. eladkal pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git. from ca4f99d Serialize the template_ext attribute to show it in UI (#17985) add 1e1b3de Fix grammar in local.rst

[GitHub] [airflow] eladkal merged pull request #18001: Correction of a slight grammatical expression error

2021-09-02 Thread GitBox
eladkal merged pull request #18001: URL: https://github.com/apache/airflow/pull/18001 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #18001: Correction of a slight grammatical expression error

2021-09-02 Thread GitBox
boring-cyborg[bot] commented on pull request #18001: URL: https://github.com/apache/airflow/pull/18001#issuecomment-912243617 Awesome work, congrats on your first merged pull request! -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [airflow] uranusjr closed pull request #17145: Adds an s3 list prefixes operator

2021-09-02 Thread GitBox
uranusjr closed pull request #17145: URL: https://github.com/apache/airflow/pull/17145 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] jarfgit opened a new pull request #17145: Adds an s3 list prefixes operator

2021-09-02 Thread GitBox
jarfgit opened a new pull request #17145: URL: https://github.com/apache/airflow/pull/17145 - Adds an operator to return a list of prefixes from an S3 bucket - Updates `list_prefixes()` unit test to assert on a nested dir with a prefix variable - Removes duplicate calls to

[GitHub] [airflow] uranusjr commented on a change in pull request #17121: [WIP] Deactivating DAGs which have been removed from files

2021-09-02 Thread GitBox
uranusjr commented on a change in pull request #17121: URL: https://github.com/apache/airflow/pull/17121#discussion_r701554837 ## File path: airflow/dag_processing/processor.py ## @@ -645,3 +648,12 @@ def process_file( self.log.exception("Error logging import

[GitHub] [airflow] uranusjr commented on pull request #17719: Change TaskInstance and TaskReschedule from execution_date to run_id

2021-09-02 Thread GitBox
uranusjr commented on pull request #17719: URL: https://github.com/apache/airflow/pull/17719#issuecomment-912219603 ``` ERROR at setup of TestSentryHook.test_add_breadcrumbs _ self = dag_maker = .DagFactory object at 0x7fadb80762e0>

[GitHub] [airflow] github-actions[bot] commented on pull request #18000: Hide variable import form if user lacks permission

2021-09-02 Thread GitBox
github-actions[bot] commented on pull request #18000: URL: https://github.com/apache/airflow/pull/18000#issuecomment-912209612 The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not

[GitHub] [airflow] uranusjr commented on pull request #17236: [Airflow 16364] Add conn_timeout and cmd_timeout params to SSHOperator; add conn_timeout param to SSHHook

2021-09-02 Thread GitBox
uranusjr commented on pull request #17236: URL: https://github.com/apache/airflow/pull/17236#issuecomment-912208597 Any further thoughts @jedcunningham? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

[GitHub] [airflow] github-actions[bot] commented on pull request #18001: Correction of a slight grammatical expression error

2021-09-02 Thread GitBox
github-actions[bot] commented on pull request #18001: URL: https://github.com/apache/airflow/pull/18001#issuecomment-912208050 The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide

[GitHub] [airflow] uranusjr commented on a change in pull request #17980: Implement API endpoint for DAG deletion

2021-09-02 Thread GitBox
uranusjr commented on a change in pull request #17980: URL: https://github.com/apache/airflow/pull/17980#discussion_r701542937 ## File path: airflow/api_connexion/endpoints/dag_endpoint.py ## @@ -100,3 +102,23 @@ def patch_dag(session, dag_id, update_mask=None):

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #18001: Correction of a slight grammatical expression error

2021-09-02 Thread GitBox
boring-cyborg[bot] commented on pull request #18001: URL: https://github.com/apache/airflow/pull/18001#issuecomment-912200402 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our

[GitHub] [airflow] FiyinD opened a new pull request #18001: Correction of a slight grammatical expression error

2021-09-02 Thread GitBox
FiyinD opened a new pull request #18001: URL: https://github.com/apache/airflow/pull/18001 --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more

[GitHub] [airflow] jedcunningham commented on a change in pull request #17980: Implement API endpoint for DAG deletion

2021-09-02 Thread GitBox
jedcunningham commented on a change in pull request #17980: URL: https://github.com/apache/airflow/pull/17980#discussion_r701511972 ## File path: airflow/api_connexion/endpoints/dag_endpoint.py ## @@ -100,3 +102,23 @@ def patch_dag(session, dag_id, update_mask=None):

[GitHub] [airflow] github-actions[bot] commented on pull request #17995: Reduce number of lines in "monitoring job" parallel group

2021-09-02 Thread GitBox
github-actions[bot] commented on pull request #17995: URL: https://github.com/apache/airflow/pull/17995#issuecomment-912156471 The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and

[GitHub] [airflow] jedcunningham opened a new pull request #18000: Hide variable import form if user lack permission

2021-09-02 Thread GitBox
jedcunningham opened a new pull request #18000: URL: https://github.com/apache/airflow/pull/18000 This hides the variable import form if the user does not have the "can create on variable" permission. Before or if the user has "can create on variable": ![Screen Shot

[GitHub] [airflow] github-actions[bot] closed issue #10686: Common tasks in downstream of multiple branches always set to skipped due to 'Not Previously Skipped' dependency

2021-09-02 Thread GitBox
github-actions[bot] closed issue #10686: URL: https://github.com/apache/airflow/issues/10686 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] github-actions[bot] commented on issue #10686: Common tasks in downstream of multiple branches always set to skipped due to 'Not Previously Skipped' dependency

2021-09-02 Thread GitBox
github-actions[bot] commented on issue #10686: URL: https://github.com/apache/airflow/issues/10686#issuecomment-912142664 This issue has been closed because it has not received response from the issue author. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] github-actions[bot] closed pull request #14640: Allow ExternalTaskSensor to wait for taskgroup

2021-09-02 Thread GitBox
github-actions[bot] closed pull request #14640: URL: https://github.com/apache/airflow/pull/14640 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] github-actions[bot] commented on pull request #17074: byte decoding to string with str method replaced by utf decoding

2021-09-02 Thread GitBox
github-actions[bot] commented on pull request #17074: URL: https://github.com/apache/airflow/pull/17074#issuecomment-912142555 This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank

[GitHub] [airflow] github-actions[bot] commented on issue #17241: Python Operator on Apple M1, python 3.9.6, airflow 2.1.2

2021-09-02 Thread GitBox
github-actions[bot] commented on issue #17241: URL: https://github.com/apache/airflow/issues/17241#issuecomment-912142532 This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further

[GitHub] [airflow] potiuk edited a comment on issue #14896: Error: 'daemonic processes are not allowed to have children', after upgradeing to airflow:2.0.1

2021-09-02 Thread GitBox
potiuk edited a comment on issue #14896: URL: https://github.com/apache/airflow/issues/14896#issuecomment-912136260 Well, I think we are talking about two different issues with "deamonic processes" and this is the source of confusion here. 1) The original stack trace here (by

[GitHub] [airflow] potiuk commented on issue #14896: Error: 'daemonic processes are not allowed to have children', after upgradeing to airflow:2.0.1

2021-09-02 Thread GitBox
potiuk commented on issue #14896: URL: https://github.com/apache/airflow/issues/14896#issuecomment-912136260 Well, I think we are talking about two different issues with "deamonic processes" and this is the source of confusion here. 1) The original stack trace here (by

[GitHub] [airflow] potiuk edited a comment on issue #17735: Permission error for error file when run_as_user is used (bug)

2021-09-02 Thread GitBox
potiuk edited a comment on issue #17735: URL: https://github.com/apache/airflow/issues/17735#issuecomment-912128633 Yeah. We have not looked at it yet, but I think this problem has a "deployment" proposal rather than "airflow" one. Simply the "other user" that you run should be

[GitHub] [airflow] potiuk commented on issue #17735: Permission error for error file when run_as_user is used (bug)

2021-09-02 Thread GitBox
potiuk commented on issue #17735: URL: https://github.com/apache/airflow/issues/17735#issuecomment-912128633 Yeah. We have not looked at it yet, but I think this problem has a "deployment" proposal rather than "airflow" one. Simply the "other user" that you run should be able to

[airflow] branch main updated: Serialize the template_ext attribute to show it in UI (#17985)

2021-09-02 Thread potiuk
This is an automated email from the ASF dual-hosted git repository. potiuk pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new ca4f99d Serialize the template_ext attribute to

[GitHub] [airflow] potiuk closed issue #17932: template_ext in task attributes shows incorrect value

2021-09-02 Thread GitBox
potiuk closed issue #17932: URL: https://github.com/apache/airflow/issues/17932 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] potiuk merged pull request #17985: Serialize the template_ext attribute to show it in UI

2021-09-02 Thread GitBox
potiuk merged pull request #17985: URL: https://github.com/apache/airflow/pull/17985 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] potiuk commented on issue #17993: Allow the default encoding to be set in airflow config to read/write task logs

2021-09-02 Thread GitBox
potiuk commented on issue #17993: URL: https://github.com/apache/airflow/issues/17993#issuecomment-912107128 Ah. I see. Apologies - this time It was also my extrapolation of what "default encoding" means :D. If we do not try to set "global default enconding" but limit it to log

[GitHub] [airflow] ashb commented on pull request #17995: Reduce number of lines in "monitoring job" parallel group

2021-09-02 Thread GitBox
ashb commented on pull request #17995: URL: https://github.com/apache/airflow/pull/17995#issuecomment-912076206 WDYT? Nothing critical, but just makes it a bit nicer to use logs view for me -- This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [airflow] ashb edited a comment on issue #14924: Scheduler Memory Leak in Airflow 2.0.1

2021-09-02 Thread GitBox
ashb edited a comment on issue #14924: URL: https://github.com/apache/airflow/issues/14924#issuecomment-912067074 When I was digging in to a similar issue I couldn't see the memory attributed to any particular process -- only the whole container via working_set_bytes -- I was

[GitHub] [airflow] ashb commented on issue #14924: Scheduler Memory Leak in Airflow 2.0.1

2021-09-02 Thread GitBox
ashb commented on issue #14924: URL: https://github.com/apache/airflow/issues/14924#issuecomment-912067074 When I was digging in to a similar issue I couldn't see the memory attributed to any particular process -- only the whole container. So that led me to believe the problem was

[GitHub] [airflow] ashb commented on issue #17833: AwsBaseHook isn't using connection.host, using connection.extra.host instead

2021-09-02 Thread GitBox
ashb commented on issue #17833: URL: https://github.com/apache/airflow/issues/17833#issuecomment-912065820 > FYI: I'm currently moving house. Should have a look at this over the weekend. My thought was leave the current functionality working as is with a deprecated warning. Then if the

[airflow] branch main updated (ab76581 -> f727635)

2021-09-02 Thread bbovenzi
This is an automated email from the ASF dual-hosted git repository. bbovenzi pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git. from ab76581 Add DAG run endpoint for marking a dagrun success or failed(#17839) add f727635 Fix blank dag

[GitHub] [airflow] bbovenzi merged pull request #17990: Fix blank dag dependencies view

2021-09-02 Thread GitBox
bbovenzi merged pull request #17990: URL: https://github.com/apache/airflow/pull/17990 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] bbovenzi closed issue #16610: Dag Dependency page not showing anything

2021-09-02 Thread GitBox
bbovenzi closed issue #16610: URL: https://github.com/apache/airflow/issues/16610 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #17998: Fix error when create external table

2021-09-02 Thread GitBox
boring-cyborg[bot] commented on pull request #17998: URL: https://github.com/apache/airflow/pull/17998#issuecomment-912055376 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our

[GitHub] [airflow] tnyz opened a new pull request #17998: Fix error when create external table

2021-09-02 Thread GitBox
tnyz opened a new pull request #17998: URL: https://github.com/apache/airflow/pull/17998 destination_project_dataset_table is being used in l1201 but cannot pass init check at 1154 --- **^ Add meaningful description above** Read the **[Pull Request

[GitHub] [airflow] alwaysmpe edited a comment on issue #17833: AwsBaseHook isn't using connection.host, using connection.extra.host instead

2021-09-02 Thread GitBox
alwaysmpe edited a comment on issue #17833: URL: https://github.com/apache/airflow/issues/17833#issuecomment-912050468 FYI: I'm currently moving house. Should have a look at this over the weekend. My thought was leave the current functionality working as is with a deprecated warning. Then

[GitHub] [airflow] jedcunningham commented on a change in pull request #17996: Update production-guide.rst - include datadog

2021-09-02 Thread GitBox
jedcunningham commented on a change in pull request #17996: URL: https://github.com/apache/airflow/pull/17996#discussion_r701417592 ## File path: docs/helm-chart/production-guide.rst ## @@ -184,6 +184,44 @@ To use an external StatsD instance: statsd_host: ...

[GitHub] [airflow] alwaysmpe commented on issue #17833: AwsBaseHook isn't using connection.host, using connection.extra.host instead

2021-09-02 Thread GitBox
alwaysmpe commented on issue #17833: URL: https://github.com/apache/airflow/issues/17833#issuecomment-912050468 FYI: I'm currently moving house. Should have a look at this over the weekend. My thought was leave the current functionality working as is, if the host value in extra isn't

[GitHub] [airflow] github-actions[bot] commented on pull request #17990: Fix blank dag dependencies view

2021-09-02 Thread GitBox
github-actions[bot] commented on pull request #17990: URL: https://github.com/apache/airflow/pull/17990#issuecomment-912043047 The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not

[GitHub] [airflow] ashb commented on a change in pull request #17719: Change TaskInstance and TaskReschedule from execution_date to run_id

2021-09-02 Thread GitBox
ashb commented on a change in pull request #17719: URL: https://github.com/apache/airflow/pull/17719#discussion_r701408236 ## File path: UPDATING.md ## @@ -228,6 +228,14 @@ Now that the DAG parser syncs DAG permissions there is no longer a need for manu In addition, the

[GitHub] [airflow] SamWheating commented on issue #17889: Airflow 2.1.3 Variable.set(name, value) statement clears out Variable Description inserted in Airflow UI

2021-09-02 Thread GitBox
SamWheating commented on issue #17889: URL: https://github.com/apache/airflow/issues/17889#issuecomment-912039643 I can put together a PR for this if you want to assign this one to me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [airflow] ashb commented on pull request #17887: New google operator: SQLToGoogleSheetsOperator

2021-09-02 Thread GitBox
ashb commented on pull request #17887: URL: https://github.com/apache/airflow/pull/17887#issuecomment-912039388 Hmmm, curious, this seems to be causing the docs build to fail on mster, but it passed on this PR branch. -- This is an automated message from the Apache Git Service. To

[GitHub] [airflow] jedcunningham commented on a change in pull request #17719: Change TaskInstance and TaskReschedule from execution_date to run_id

2021-09-02 Thread GitBox
jedcunningham commented on a change in pull request #17719: URL: https://github.com/apache/airflow/pull/17719#discussion_r701396620 ## File path: UPDATING.md ## @@ -228,6 +228,14 @@ Now that the DAG parser syncs DAG permissions there is no longer a need for manu In

[GitHub] [airflow] suhanovv commented on issue #14924: Scheduler Memory Leak in Airflow 2.0.1

2021-09-02 Thread GitBox
suhanovv commented on issue #14924: URL: https://github.com/apache/airflow/issues/14924#issuecomment-912028969 @potiuk At the moment, the container_memory_working_set_bytes has grown by about 150-200 mb, I will observe and try to further investigate the reason for this behavior. I

[GitHub] [airflow] SamWheating commented on a change in pull request #17121: [WIP] Deactivating DAGs which have been removed from files

2021-09-02 Thread GitBox
SamWheating commented on a change in pull request #17121: URL: https://github.com/apache/airflow/pull/17121#discussion_r701386688 ## File path: airflow/dag_processing/processor.py ## @@ -645,3 +648,12 @@ def process_file( self.log.exception("Error logging import

[GitHub] [airflow] a-nascimento opened a new pull request #17996: Update production-guide.rst - include datadog

2021-09-02 Thread GitBox
a-nascimento opened a new pull request #17996: URL: https://github.com/apache/airflow/pull/17996 This is to enable exporting of logs to a datadog agent running in the same cluster. --- **^ Add meaningful description above** Read the **[Pull Request

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #17996: Update production-guide.rst - include datadog

2021-09-02 Thread GitBox
boring-cyborg[bot] commented on pull request #17996: URL: https://github.com/apache/airflow/pull/17996#issuecomment-912019305 Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our

[GitHub] [airflow] SamWheating commented on a change in pull request #17121: [WIP] Deactivating DAGs which have been removed from files

2021-09-02 Thread GitBox
SamWheating commented on a change in pull request #17121: URL: https://github.com/apache/airflow/pull/17121#discussion_r701386688 ## File path: airflow/dag_processing/processor.py ## @@ -645,3 +648,12 @@ def process_file( self.log.exception("Error logging import

[GitHub] [airflow] SamWheating commented on a change in pull request #17121: [WIP] Deactivating DAGs which have been removed from files

2021-09-02 Thread GitBox
SamWheating commented on a change in pull request #17121: URL: https://github.com/apache/airflow/pull/17121#discussion_r701386688 ## File path: airflow/dag_processing/processor.py ## @@ -645,3 +648,12 @@ def process_file( self.log.exception("Error logging import

[GitHub] [airflow] Narendra-Neerukonda edited a comment on issue #17993: Allow the default encoding to be set in airflow config to read/write task logs

2021-09-02 Thread GitBox
Narendra-Neerukonda edited a comment on issue #17993: URL: https://github.com/apache/airflow/issues/17993#issuecomment-912012496 Ahh... All well from my end too @potiuk Very thankful for your patience in explaining the details above. I understood what you are trying to convey. I

[GitHub] [airflow] Narendra-Neerukonda commented on issue #17993: Allow the default encoding to be set in airflow config to read/write task logs

2021-09-02 Thread GitBox
Narendra-Neerukonda commented on issue #17993: URL: https://github.com/apache/airflow/issues/17993#issuecomment-912012496 Ahh... All well from my end too @potiuk Very thankful for your patience in explaining the details above. I was just trying to add a config like say

[GitHub] [airflow] siddharthvp commented on a change in pull request #17121: [WIP] Deactivating DAGs which have been removed from files

2021-09-02 Thread GitBox
siddharthvp commented on a change in pull request #17121: URL: https://github.com/apache/airflow/pull/17121#discussion_r701382534 ## File path: airflow/dag_processing/processor.py ## @@ -645,3 +648,12 @@ def process_file( self.log.exception("Error logging import

[GitHub] [airflow] potiuk edited a comment on issue #14924: Scheduler Memory Leak in Airflow 2.0.1

2021-09-02 Thread GitBox
potiuk edited a comment on issue #14924: URL: https://github.com/apache/airflow/issues/14924#issuecomment-912003732 @suhanovv - Is the `container_memory_working_set_bytes` growing still continuously now ? Can you please observe it for a while and when it grows, compare the memory used by

[GitHub] [airflow] potiuk edited a comment on issue #14924: Scheduler Memory Leak in Airflow 2.0.1

2021-09-02 Thread GitBox
potiuk edited a comment on issue #14924: URL: https://github.com/apache/airflow/issues/14924#issuecomment-912003732 @suhanovv - Is the `container_memory_working_set_bytes` growing still continuously now ? Can you please observe it for a while and when it grows, compare the memory used by

[GitHub] [airflow] potiuk commented on issue #14924: Scheduler Memory Leak in Airflow 2.0.1

2021-09-02 Thread GitBox
potiuk commented on issue #14924: URL: https://github.com/apache/airflow/issues/14924#issuecomment-912003732 @suhanovv - Is the `container_memory_working_set_bytes` growing still continuously now ? Can you observe it for a while and when it grows, compare the memory used by processes and

[GitHub] [airflow] ashb opened a new pull request #17995: Reduce number of lines in "monitoring job" parallel group

2021-09-02 Thread GitBox
ashb opened a new pull request #17995: URL: https://github.com/apache/airflow/pull/17995 Although this group is collapsed when the job is finished it it is large it can cause the logs view in GitHub to bog down the browser, so this PR does a few things: - Stops showing disk

[GitHub] [airflow] potiuk commented on pull request #17944: Adds example showing the ES_hook

2021-09-02 Thread GitBox
potiuk commented on pull request #17944: URL: https://github.com/apache/airflow/pull/17944#issuecomment-911996570 爛 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [airflow] boring-cyborg[bot] commented on pull request #17839: Add DAG run abort endpoint

2021-09-02 Thread GitBox
boring-cyborg[bot] commented on pull request #17839: URL: https://github.com/apache/airflow/pull/17839#issuecomment-911993338 Awesome work, congrats on your first merged pull request! -- This is an automated message from the Apache Git Service. To respond to the message, please log

[airflow] branch main updated: Add DAG run endpoint for marking a dagrun success or failed(#17839)

2021-09-02 Thread ephraimanierobi
This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/main by this push: new ab76581 Add DAG run endpoint for marking

[GitHub] [airflow] ephraimbuddy merged pull request #17839: Add DAG run abort endpoint

2021-09-02 Thread GitBox
ephraimbuddy merged pull request #17839: URL: https://github.com/apache/airflow/pull/17839 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] ephraimbuddy closed issue #15888: Abort a DAG Run

2021-09-02 Thread GitBox
ephraimbuddy closed issue #15888: URL: https://github.com/apache/airflow/issues/15888 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] potiuk commented on issue #17993: Allow the default encoding to be set in airflow config to read/write task logs

2021-09-02 Thread GitBox
potiuk commented on issue #17993: URL: https://github.com/apache/airflow/issues/17993#issuecomment-911985020 It's nothing personal @Narendra-Neerukonda - It's just comment that I think what you proposed (reading from config and making sure that it is used for all reads/writes by setting

[GitHub] [airflow] christine-le commented on issue #15607: airflow throws error: airflow.exceptions.SerializedDagNotFound: DAG 'dns-basic-1min' not found in serialized_dag table

2021-09-02 Thread GitBox
christine-le commented on issue #15607: URL: https://github.com/apache/airflow/issues/15607#issuecomment-911984237 Just wanted to chime in to say we ran into the exact same error under pretty similar set up as @neilharia7 with a set of dynamic dags. Setting processor_poll_interval to 30

[GitHub] [airflow] potiuk edited a comment on issue #17993: Allow the default encoding to be set in airflow config to read/write task logs

2021-09-02 Thread GitBox
potiuk edited a comment on issue #17993: URL: https://github.com/apache/airflow/issues/17993#issuecomment-911978583 I never said anything about reading default encoding from `airflow.cfg` and set default encoding based on that: > It would be great to standardise encoding to "utf-8"

[GitHub] [airflow] potiuk edited a comment on issue #17993: Allow the default encoding to be set in airflow config to read/write task logs

2021-09-02 Thread GitBox
potiuk edited a comment on issue #17993: URL: https://github.com/apache/airflow/issues/17993#issuecomment-911978583 I never said anything about reading default encoding from `airflow.cfg` and set default encoding based on that: > It would be great to standardise encoding to "utf-8"

[GitHub] [airflow] potiuk edited a comment on issue #17993: Allow the default encoding to be set in airflow config to read/write task logs

2021-09-02 Thread GitBox
potiuk edited a comment on issue #17993: URL: https://github.com/apache/airflow/issues/17993#issuecomment-911978583 I never said anything about reading default encoding from `airflow.cfg` and set default encoding based on that: > It would be great to standardise encoding to "utf-8"

[GitHub] [airflow] potiuk edited a comment on issue #17993: Allow the default encoding to be set in airflow config to read/write task logs

2021-09-02 Thread GitBox
potiuk edited a comment on issue #17993: URL: https://github.com/apache/airflow/issues/17993#issuecomment-911978583 I never said anything about reading default encoding from `airflow.cfg` and set default encoding based on that: > It would be great to standardise encoding to "utf-8"

[GitHub] [airflow] potiuk edited a comment on issue #17993: Allow the default encoding to be set in airflow config to read/write task logs

2021-09-02 Thread GitBox
potiuk edited a comment on issue #17993: URL: https://github.com/apache/airflow/issues/17993#issuecomment-911978583 I never said anything about reading default encoding from `airflow.cfg` and set default encoding: > It would be great to standardise encoding to "utf-8" (default)

[GitHub] [airflow] potiuk edited a comment on issue #17993: Allow the default encoding to be set in airflow config to read/write task logs

2021-09-02 Thread GitBox
potiuk edited a comment on issue #17993: URL: https://github.com/apache/airflow/issues/17993#issuecomment-911978583 I never said anything about reading default encoding in `airflow.cfg` and set default encoding: > It would be great to standardise encoding to "utf-8" (default) when

[GitHub] [airflow] potiuk edited a comment on issue #17993: Allow the default encoding to be set in airflow config to read/write task logs

2021-09-02 Thread GitBox
potiuk edited a comment on issue #17993: URL: https://github.com/apache/airflow/issues/17993#issuecomment-911978583 I never said anythint to reading default encoding in `airflow.cfg` and set default encoding: > It would be great to standardise encoding to "utf-8" (default) when

[GitHub] [airflow] potiuk commented on issue #17993: Allow the default encoding to be set in airflow config to read/write task logs

2021-09-02 Thread GitBox
potiuk commented on issue #17993: URL: https://github.com/apache/airflow/issues/17993#issuecomment-911978583 I never said anythint to reading default encoding in `airflow.cfg` and set default encoding: > It would be great to standardise encoding to "utf-8" (default) when writing

[GitHub] [airflow] Narendra-Neerukonda commented on issue #17993: Allow the default encoding to be set in airflow config to read/write task logs

2021-09-02 Thread GitBox
Narendra-Neerukonda commented on issue #17993: URL: https://github.com/apache/airflow/issues/17993#issuecomment-911975711 Hi @potiuk This feature request is in reference to your comment: https://github.com/apache/airflow/issues/16834#issuecomment-877607117 Can you please once

[GitHub] [airflow] SamWheating commented on a change in pull request #17121: [WIP] Deactivating DAGs which have been removed from files

2021-09-02 Thread GitBox
SamWheating commented on a change in pull request #17121: URL: https://github.com/apache/airflow/pull/17121#discussion_r701344776 ## File path: airflow/dag_processing/processor.py ## @@ -645,3 +648,12 @@ def process_file( self.log.exception("Error logging import

[GitHub] [airflow] rpfernandezjr commented on issue #15580: KubernetesExecutor not deleting worker pods on completion.

2021-09-02 Thread GitBox
rpfernandezjr commented on issue #15580: URL: https://github.com/apache/airflow/issues/15580#issuecomment-911966784 @kipkoan - no i just ended up using `restartPolicy: Never` in my pod template spec. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [airflow] potiuk edited a comment on issue #17993: Allow the default encoding to be set in airflow config to read/write task logs

2021-09-02 Thread GitBox
potiuk edited a comment on issue #17993: URL: https://github.com/apache/airflow/issues/17993#issuecomment-911960059 @Narendra-Neerukonda My point is that I believe you "CAN'T" reliably set encoding in Python code in all scenarios. Once Python interpreter starts, you are basically done

[GitHub] [airflow] potiuk commented on issue #17993: Allow the default encoding to be set in airflow config to read/write task logs

2021-09-02 Thread GitBox
potiuk commented on issue #17993: URL: https://github.com/apache/airflow/issues/17993#issuecomment-911960059 @Narendra-Neerukonda My point is that I believe you "CAN'T" reliably set encoding in Python code in all scenarios. Once Python interpreter starts, you are basically done (and this

[GitHub] [airflow] potiuk commented on issue #17897: Dag tags are not refreshing if case sensitivity changed

2021-09-02 Thread GitBox
potiuk commented on issue #17897: URL: https://github.com/apache/airflow/issues/17897#issuecomment-911951115 Very, very interesting. Curious to see it with the fix. Is it also possible that you add two wanings to those two sides of the condition and see the Warnings generated? (there

[GitHub] [airflow] potiuk commented on issue #10285: Update example DAG files to latest syntax

2021-09-02 Thread GitBox
potiuk commented on issue #10285: URL: https://github.com/apache/airflow/issues/10285#issuecomment-911947263 Fantastic! Thanks @josh-fell ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] dimon222 edited a comment on issue #17897: Dag tags are not refreshing if case sensitivity changed

2021-09-02 Thread GitBox
dimon222 edited a comment on issue #17897: URL: https://github.com/apache/airflow/issues/17897#issuecomment-911947035 Just tested on Mysql 8.0.11 and was able to replicate exactly same scenario. As reminder, 1. Store DAGs to True 2. Create dag with tags, let airflow pick it up

[GitHub] [airflow] dimon222 commented on issue #17897: Dag tags are not refreshing if case sensitivity changed

2021-09-02 Thread GitBox
dimon222 commented on issue #17897: URL: https://github.com/apache/airflow/issues/17897#issuecomment-911947035 Just tested on Mysql 8.0.11 and was able to replicate exactly same scenario. As reminder, 1. Store DAGs to True 2. Create dag with tags, let airflow pick it up 3.

[GitHub] [airflow] mik-laj commented on issue #10285: Update example DAG files to latest syntax

2021-09-02 Thread GitBox
mik-laj commented on issue #10285: URL: https://github.com/apache/airflow/issues/10285#issuecomment-911944188 @josh-fell thanks. Closed! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

[GitHub] [airflow] mik-laj closed issue #10285: Update example DAG files to latest syntax

2021-09-02 Thread GitBox
mik-laj closed issue #10285: URL: https://github.com/apache/airflow/issues/10285 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [airflow] Narendra-Neerukonda edited a comment on issue #17993: Allow the default encoding to be set in airflow config to read/write task logs

2021-09-02 Thread GitBox
Narendra-Neerukonda edited a comment on issue #17993: URL: https://github.com/apache/airflow/issues/17993#issuecomment-911916814 @potiuk , As of now i was thinking to allow users to specify the desired encoding for task logs in airflow.cfg and use utf-8 as default in case users don't

[GitHub] [airflow] Narendra-Neerukonda edited a comment on issue #17993: Allow the default encoding to be set in airflow config to read/write task logs

2021-09-02 Thread GitBox
Narendra-Neerukonda edited a comment on issue #17993: URL: https://github.com/apache/airflow/issues/17993#issuecomment-911916814 @potiuk , As of now i was thinking to allow users to specify the desired encoding for task logs in airflow.cfg and use utf-8 as default in case users don't

[GitHub] [airflow] Narendra-Neerukonda edited a comment on issue #17993: Allow the default encoding to be set in airflow config to read/write task logs

2021-09-02 Thread GitBox
Narendra-Neerukonda edited a comment on issue #17993: URL: https://github.com/apache/airflow/issues/17993#issuecomment-911916814 @potiuk , I'll need some time to research on this to find the best solution. As of now i was thinking to allow users to specify the desired encoding for task

[GitHub] [airflow] Narendra-Neerukonda edited a comment on issue #17993: Allow the default encoding to be set in airflow config to read/write task logs

2021-09-02 Thread GitBox
Narendra-Neerukonda edited a comment on issue #17993: URL: https://github.com/apache/airflow/issues/17993#issuecomment-911916814 @potiuk , I'll need some time to research on this to find the best solution. As of now i was thinking to allow users to specify the desired encoding for task

[GitHub] [airflow] Narendra-Neerukonda commented on issue #17993: Allow the default encoding to be set in airflow config to read/write task logs

2021-09-02 Thread GitBox
Narendra-Neerukonda commented on issue #17993: URL: https://github.com/apache/airflow/issues/17993#issuecomment-911916814 @potiuk , I'll need some time to research on this to find the best solution. As of now i was thinking to allow users to specify the desired encoding for task logs

[GitHub] [airflow] suhanovv edited a comment on issue #14924: Scheduler Memory Leak in Airflow 2.0.1

2021-09-02 Thread GitBox
suhanovv edited a comment on issue #14924: URL: https://github.com/apache/airflow/issues/14924#issuecomment-911914984 > > my screenshot shows the metric container_memory_working_set_bytes, it decreased after 10 seconds (scraping time of metrics) after rm execution, scheduler did not

[GitHub] [airflow] suhanovv commented on issue #14924: Scheduler Memory Leak in Airflow 2.0.1

2021-09-02 Thread GitBox
suhanovv commented on issue #14924: URL: https://github.com/apache/airflow/issues/14924#issuecomment-911914984 > > my screenshot shows the metric container_memory_working_set_bytes, it decreased after 10 seconds (scraping time of metrics) after rm execution, scheduler did not restart >

[GitHub] [airflow] Narendra-Neerukonda commented on a change in pull request #17965: sets encoding to utf-8 by default while reading task logs

2021-09-02 Thread GitBox
Narendra-Neerukonda commented on a change in pull request #17965: URL: https://github.com/apache/airflow/pull/17965#discussion_r701211786 ## File path: airflow/utils/log/file_task_handler.py ## @@ -116,7 +116,7 @@ def _read(self, ti, try_number, metadata=None): if

[GitHub] [airflow] Narendra-Neerukonda removed a comment on issue #17993: Allow the default encoding to be set in airflow config to read/write task logs

2021-09-02 Thread GitBox
Narendra-Neerukonda removed a comment on issue #17993: URL: https://github.com/apache/airflow/issues/17993#issuecomment-911823175 i would like to work on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

  1   2   3   4   >