Re: [PR] Change capture warnings output format to the JSON [airflow]

[via GitHub]

Taragolis merged PR #38905:
URL: https://github.com/apache/airflow/pull/38905


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Change capture warnings output format to the JSON [airflow]

[via GitHub]

Taragolis commented on PR #38905:
URL: https://github.com/apache/airflow/pull/38905#issuecomment-2057430746

   Feel difference
   
   **Python 3.12 (non-db)**
   ```console
   == Warning summary. Total: 4,670, Unique: 329 
==
   airflow: total 133, unique 16
   other: total 3,737, unique 24
   providers: total 222, unique 44
   tests: total 578, unique 245
   Warnings saved into /files/warnings-all-none.txt file.
   ```
   
   **Python 3.11 (non-db)**
   ```console
   == Warning summary. Total: 1,136, Unique: 320 
==
   airflow: total 132, unique 15
   other: total 204, unique 17
   providers: total 224, unique 45
   tests: total 576, unique 243
   Warnings saved into /files/warnings-all-none.txt file.
   ```
   
   **Python 3.8 (non-db)**
   ```console
   == Warning summary. Total: 1,134, Unique: 320 
==
   airflow: total 132, unique 15
   other: total 203, unique 18
   providers: total 223, unique 44
   tests: total 576, unique 243
   Warnings saved into /files/warnings-all-none.txt file.
   ```
   
   The differences mostly because some upstream dependency still use 
`datetime.datetime.utcnow()` (e.g. botocore), some deprecated constant from ast 
module (werkzeug, should be resolved as soon as we will able to update to newer 
version)
   
   And also fork from multithreading, which are "deprecated" in Python 3.12: 
https://discuss.python.org/t/concerns-regarding-deprecation-of-fork-with-alive-threads/33555
   
   ```console
   {"category": "DeprecationWarning", "message": "This process (pid=99) is 
multi-threaded, use of fork() may lead to deadlocks in the child.", "node_id": 
"tests/secrets/test_cache.py::TestSecretCache::test_cache_accessible_from_other_process",
 "filename": "/usr/local/lib/python3.12/multiprocessing/popen_fork.py", 
"lineno": 66}
   {"category": "DeprecationWarning", "message": "This process (pid=99) is 
multi-threaded, use of fork() may lead to deadlocks in the child.", "node_id": 
"tests/providers/celery/executors/test_celery_kubernetes_executor.py::TestCeleryKubernetesExecutor::test_queue_command",
 "filename": "/usr/local/lib/python3.12/multiprocessing/popen_fork.py", 
"lineno": 66}
   {"category": "DeprecationWarning", "message": "This process (pid=99) is 
multi-threaded, use of fork() may lead to deadlocks in the child.", "node_id": 
"tests/providers/celery/executors/test_celery_kubernetes_executor.py::TestCeleryKubernetesExecutor::test_job_id_setter",
 "filename": "/usr/local/lib/python3.12/multiprocessing/popen_fork.py", 
"lineno": 66}
   {"category": "DeprecationWarning", "message": "This process (pid=99) is 
multi-threaded, use of fork() may lead to deadlocks in the child.", "node_id": 
"tests/providers/celery/executors/test_celery_kubernetes_executor.py::TestCeleryKubernetesExecutor::test_send_callback",
 "filename": "/usr/local/lib/python3.12/multiprocessing/popen_fork.py", 
"lineno": 66}
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Change capture warnings output format to the JSON [airflow]

[via GitHub]

vincbeck commented on code in PR #38905:
URL: https://github.com/apache/airflow/pull/38905#discussion_r1565766792


##
contributing-docs/testing/unit_tests.rst:
##
@@ -1140,6 +1140,34 @@ for prevent to run on unsupported platform.
 - ``breeze``: Run test only inside of Breeze container, it might be useful in 
case of run
   some potential dangerous things in tests or if it expects to use common 
Breeze things.
 
+Warnings capture system
+...
+
+By default all warnings which captured during tests run saved into the 
``tests/warnings.txt``.

Review Comment:
   ```suggestion
   By default, all warnings captured during the test runs are saved into the 
``tests/warnings.txt``.
   ```



##
contributing-docs/testing/unit_tests.rst:
##
@@ -1140,6 +1140,34 @@ for prevent to run on unsupported platform.
 - ``breeze``: Run test only inside of Breeze container, it might be useful in 
case of run
   some potential dangerous things in tests or if it expects to use common 
Breeze things.
 
+Warnings capture system
+...
+
+By default all warnings which captured during tests run saved into the 
``tests/warnings.txt``.
+
+If required you could change the path by providing ``--warning-output-path`` 
in pytest CLI arguments
+or by set environment variable ``CAPTURE_WARNINGS_OUTPUT``.
+
+.. code-block:: console
+
+root@3f98e75b1ebe:/opt/airflow# pytest tests/core/ 
--warning-output-path=/foo/bar/spam.egg
+...
+= Warning summary. Total: 34, Unique: 16 
==
+airflow: total 11, unique 1
+other: total 12, unique 4
+tests: total 11, unique 11
+Warnings saved into /foo/bar/spam.egg file.
+
+= short test summary info 
=
+
+You might also disable capture system by provide 
``--disable-capture-warnings`` in pytest CLI arguments
+or by by set `global warnings filter 
`__

Review Comment:
   ```suggestion
   or by setting `global warnings filter 
`__
   ```



##
contributing-docs/testing/unit_tests.rst:
##
@@ -1140,6 +1140,34 @@ for prevent to run on unsupported platform.
 - ``breeze``: Run test only inside of Breeze container, it might be useful in 
case of run
   some potential dangerous things in tests or if it expects to use common 
Breeze things.
 
+Warnings capture system
+...
+
+By default all warnings which captured during tests run saved into the 
``tests/warnings.txt``.
+
+If required you could change the path by providing ``--warning-output-path`` 
in pytest CLI arguments
+or by set environment variable ``CAPTURE_WARNINGS_OUTPUT``.
+
+.. code-block:: console
+
+root@3f98e75b1ebe:/opt/airflow# pytest tests/core/ 
--warning-output-path=/foo/bar/spam.egg
+...
+= Warning summary. Total: 34, Unique: 16 
==
+airflow: total 11, unique 1
+other: total 12, unique 4
+tests: total 11, unique 11
+Warnings saved into /foo/bar/spam.egg file.
+
+= short test summary info 
=
+
+You might also disable capture system by provide 
``--disable-capture-warnings`` in pytest CLI arguments
+or by by set `global warnings filter 
`__
+to **ignore**, e.g. by set ``PYTHONWARNINGS`` environment variable to 
``ignore``.

Review Comment:
   ```suggestion
   to **ignore**, e.g. set ``PYTHONWARNINGS`` environment variable to 
``ignore``.
   ```



##
contributing-docs/testing/unit_tests.rst:
##
@@ -1140,6 +1140,34 @@ for prevent to run on unsupported platform.
 - ``breeze``: Run test only inside of Breeze container, it might be useful in 
case of run
   some potential dangerous things in tests or if it expects to use common 
Breeze things.
 
+Warnings capture system
+...
+
+By default all warnings which captured during tests run saved into the 
``tests/warnings.txt``.
+
+If required you could change the path by providing ``--warning-output-path`` 
in pytest CLI arguments

Review Comment:
   ```suggestion
   If required, you could change the path by providing 
``--warning-output-path`` as pytest CLI arguments
   ```



##
contributing-docs/testing/unit_tests.rst:
##
@@ -1140,6 +1140,34 @@ for prevent to run on unsupported platform.
 - ``breeze``: Run test only inside of Breeze container, it might be useful in 
case of run
   some potential dangerous things in tests or if it expects to use common 
Breeze things.
 
+Warnings capture system
+...
+
+By default all warnings which captured during tests run saved into the 
``tests/warnings.txt``.
+
+If required you could change the path by providing 

Re: [PR] Change capture warnings output format to the JSON [airflow]

[via GitHub]

Taragolis commented on PR #38905:
URL: https://github.com/apache/airflow/pull/38905#issuecomment-2049994033

   Covert to draft for avoid accidentally merge
   
   I have a plan to change a bit what we store into output file
   Pytest output format, e.g. where warning raised airflow, providers, test or 
other
   Add warning summariser in the CI
   
   Warnings output from this PR enough for play with merge into resulting dump, 
split into the categories and found some unnecessary or important warnings


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Change capture warnings output format to the JSON [airflow]

[via GitHub]

Taragolis commented on PR #38905:
URL: https://github.com/apache/airflow/pull/38905#issuecomment-2048455211

   Also add about configuration into the documentation, so worthwhile to check 
spelling and grammar 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Change capture warnings output format to the JSON [airflow]

[via GitHub]

Taragolis commented on PR #38905:
URL: https://github.com/apache/airflow/pull/38905#issuecomment-2048332619

   Test 
`tests/ti_deps/deps/test_mapped_task_upstream_dep.py::test_step_by_step` is so 
annoying, it failed into the CI on the regular basis in the different PR and I 
can't still reproduce it locally 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Change capture warnings output format to the JSON [airflow]

[via GitHub]

Taragolis closed pull request #38905: Change capture warnings output format to 
the JSON
URL: https://github.com/apache/airflow/pull/38905


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Change capture warnings output format to the JSON [airflow]

[via GitHub]

mergeable[bot] commented on PR #38905:
URL: https://github.com/apache/airflow/pull/38905#issuecomment-2048218996

     :x: Error Occurred while executing an Action 
   
If you believe this is an unexpected error, please report it on our issue 
tracker: https://github.com/mergeability/mergeable/issues/new 
   # Error Details 
    
   HttpError: Server Error


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Change capture warnings output format to the JSON [airflow]

[via GitHub]

mergeable[bot] commented on PR #38905:
URL: https://github.com/apache/airflow/pull/38905#issuecomment-2048215410

     :x: Error Occurred while executing an Action 
   
If you believe this is an unexpected error, please report it on our issue 
tracker: https://github.com/mergeability/mergeable/issues/new 
   # Error Details 
    
   HttpError: Server Error


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] Change capture warnings output format to the JSON [airflow]

[via GitHub]

Taragolis opened a new pull request, #38905:
URL: https://github.com/apache/airflow/pull/38905

   
   
   
   
   This PR is a result of complete rework on warning capture system:
   - Pass parameters from the pytest configurations
   - Trim path to the project root or sites
   - Keep information about original warning and in which test suite it 
originally raised
   - Avoid to use internal pytests classes
   - Output format is JSON rather than pylint (we do not use it anyway). 
Drawbacks is CI warnings summary might show greater number, it could be fixed 
as a follow up by implement simple summarised python script.
   - Output path now relative to the tests directory, but default it stored 
into the `tests/warnings.txt` regardless of the current working directory
   
   The only things the same with 
[pytest-capture-warnings](https://github.com/athinkingape/pytest-capture-warnings)
 is an idea, so I remove license headers, if some one have any objections feel 
free to ping me to return it. 
   
   **Output before**
   
   ```console
   tests/core/test_configuration.py:1411: [W0513(warning), ] The 
sql_alchemy_conn option in [core] has been moved to the sql_alchemy_conn option 
in [database] - the old setting has been used, but please update your config.
   tests/core/test_core.py:107: [W0513(warning), ] SELECT statement has a 
cartesian product between FROM element(s) "task_fail" and FROM element 
"dag_run".  Apply join condition(s) between each element to resolve.
   tests/core/test_logging_config.py:267: [W0513(warning), ] The 
`airflow.utils.log.wasb_task_handler.WasbTaskHandler` class is deprecated. 
Please use 
`'airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler'`.
   ```
   
   **Output after**
   ```console
   {"category": "DeprecationWarning", "message": "The sql_alchemy_conn option 
in [core] has been moved to the sql_alchemy_conn option in [database] - the old 
setting has been used, but please update your config.", "node_id": 
"tests/core/test_configuration.py::TestDeprecatedConf::test_conf_as_dict_when_deprecated_value_in_secrets",
 "filename": "tests/core/test_configuration.py", "lineno": 1411, "count": 1}
   {"category": "sqlalchemy.exc.SAWarning", "message": "SELECT statement has a 
cartesian product between FROM element(s) \"task_fail\" and FROM element 
\"dag_run\".  Apply join condition(s) between each element to resolve.", 
"node_id": "tests/core/test_core.py::TestCore::test_task_fail_duration", 
"filename": "tests/core/test_core.py", "lineno": 107, "count": 1}
   {"category": "DeprecationWarning", "message": "The 
`airflow.utils.log.wasb_task_handler.WasbTaskHandler` class is deprecated. 
Please use 
`'airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler'`.", 
"node_id": 
"tests/core/test_logging_config.py::TestLoggingSettings::test_loading_remote_logging_with_wasb_handler",
 "filename": "tests/core/test_logging_config.py", "lineno": 267, "count": 1}
   ```
   
   
   ---
   **^ Add meaningful description above**
   Read the **[Pull Request 
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
 for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal 
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
 is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party 
License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a 
newsfragment file, named `{pr_number}.significant.rst` or 
`{issue_number}.significant.rst`, in 
[newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org