Re: [PATCH] Ignore flaky ob-python tests

[Ihor Radchenko]

Christian Köstlin  writes:

> I somehow seem to remember that the async tests were also broken with
> python.el (python-mode.el not installed).
> at least sometimes?

I only recall the underscore test.

Note that I debugged the async tests with your rake builds, and I have
identified that async tests are failing because of python-mode
inclusion. (python-mode is not supported by async ob-python code).

> or is this only the underscore test?
> If it's only the underscore test, I would suggest not installing
> python-mode for ci and ignoring the really flaky tests (i think the
> underscore one is a candidate here).

Yes. Please, go ahead and remove python-mode from the CI tests.

We are likely going to deprecate python-mode support. See
https://orgmode.org/list/87r0yk7bx8.fsf@localhost

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [PATCH] Ignore flaky ob-python tests

[Christian Köstlin]

Hi Ihor,

I somehow seem to remember that the async tests were also broken with
python.el (python-mode.el not installed).
at least sometimes? or is this only the underscore test?
If it's only the underscore test, I would suggest not installing
python-mode for ci and ignoring the really flaky tests (i think the
underscore one is a candidate here).

I somehow got the async things mixed up with the flaky tests, which is
not true for python-mode.el installed.

what do you think?

I really would like to see python tests pass ...

Kind regards,
Christian

On Thu, Nov 3, 2022 at 8:13 AM Ihor Radchenko  wrote:
>
> Christian Köstlin  writes:
>
> > With my patch the tests are still failing but the testsuite continues
> > to run and returns
> > with 0 as status code (one can see the failed test though).
>
> FYI, I am using automated scripts to run make test locally on multiple
> Emacs versions. 0 status code would do no good for my testing.
>
> >> Maybe we can instead provide some variable that can turn-on ignoring
> >> some problematic tests only during CI tests?
> > Yes ... that would also be possible e.g. skip the test or test nothing
> > if it's running on the ci.
> > (I am talking only about the 3 tests!).
>
> More like just 1 test with underscore :)
> The 3 tests in the patch are real failures because ob-python does not
> fully support python-mode.el.
>
> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at .
> Support Org development at ,
> or support my work at 

Re: [PATCH] Ignore flaky ob-python tests

[Ihor Radchenko]

Christian Köstlin  writes:

> With my patch the tests are still failing but the testsuite continues
> to run and returns
> with 0 as status code (one can see the failed test though).

FYI, I am using automated scripts to run make test locally on multiple
Emacs versions. 0 status code would do no good for my testing.

>> Maybe we can instead provide some variable that can turn-on ignoring
>> some problematic tests only during CI tests?
> Yes ... that would also be possible e.g. skip the test or test nothing
> if it's running on the ci.
> (I am talking only about the 3 tests!).

More like just 1 test with underscore :)
The 3 tests in the patch are real failures because ob-python does not
fully support python-mode.el.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [PATCH] Ignore flaky ob-python tests

[Ihor Radchenko]

Ihor Radchenko  writes:

> Jack Kamm  writes:
>
>> Christian Köstlin  writes:
>>
>>> I think we know of the async problem, and you reported that as well,
>>> so I would expect
>>> if someone is able to fix it, he will also move the tests to "sharp" again.
>>
>> Could you provide some references about the async problem, either how to
>> reproduce it or the underlying cause? I tried searching emacs-orgmode
>> and emacs-devel, but couldn't find a report of the problem.

Also, 
https://lists.sr.ht/~bzg/org-build-failures/%3CCNZYPX9R5ABN.VI1C53TZJSTI%40cirno2%3E

BTW, Christian, failures of tests in the patch are valid failures that
revealed a real problem.
See https://orgmode.org/list/87bkprid1d.fsf@localhost

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [PATCH] Ignore flaky ob-python tests

[Ihor Radchenko]

Jack Kamm  writes:

> Christian Köstlin  writes:
>
>> I think we know of the async problem, and you reported that as well,
>> so I would expect
>> if someone is able to fix it, he will also move the tests to "sharp" again.
>
> Could you provide some references about the async problem, either how to
> reproduce it or the underlying cause? I tried searching emacs-orgmode
> and emacs-devel, but couldn't find a report of the problem.

https://lists.sr.ht/~bzg/org-build-failures/%3CCNRV9YIXSILF.2AF0TMJG974H4%40cirno%3E


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

Re: [PATCH] Ignore flaky ob-python tests

[Jack Kamm]

Christian Köstlin  writes:

> I think we know of the async problem, and you reported that as well,
> so I would expect
> if someone is able to fix it, he will also move the tests to "sharp" again.

Could you provide some references about the async problem, either how to
reproduce it or the underlying cause? I tried searching emacs-orgmode
and emacs-devel, but couldn't find a report of the problem.

Re: [PATCH] Ignore flaky ob-python tests

[Christian Köstlin]

On Wed, Nov 2, 2022 at 7:52 AM Ihor Radchenko  wrote:
>
> Christian Köstlin  writes:
>
> > The new async feature of python seems to be implemented in a way
> > that can break with high CPU load. This patch still runs the 3 tests in
> > question, but does not fail the complete testsuite if any of those fail.
>
> Thanks for the patch, but I'm afraid that it will drive the tests useless.
> After the patch, even if the tests fail for a valid reason, it will not
> be visible upon running make test, AFAIU.
I think we know of the async problem, and you reported that as well,
so I would expect
if someone is able to fix it, he will also move the tests to "sharp" again.
With my patch the tests are still failing but the testsuite continues
to run and returns
with 0 as status code (one can see the failed test though).

> Maybe we can instead provide some variable that can turn-on ignoring
> some problematic tests only during CI tests?
Yes ... that would also be possible e.g. skip the test or test nothing
if it's running on the ci.
(I am talking only about the 3 tests!).

Kind regards,
Christian

Re: [PATCH] Ignore flaky ob-python tests

[Ihor Radchenko]

Christian Köstlin  writes:

> The new async feature of python seems to be implemented in a way
> that can break with high CPU load. This patch still runs the 3 tests in
> question, but does not fail the complete testsuite if any of those fail.

Thanks for the patch, but I'm afraid that it will drive the tests useless.
After the patch, even if the tests fail for a valid reason, it will not
be visible upon running make test, AFAIU.

Maybe we can instead provide some variable that can turn-on ignoring
some problematic tests only during CI tests?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at .
Support Org development at ,
or support my work at 

[PATCH] Ignore flaky ob-python tests

[Christian Köstlin]

The new async feature of python seems to be implemented in a way
that can break with high CPU load. This patch still runs the 3 tests in
question, but does not fail the complete testsuite if any of those fail.

Kind regards,
Christian


0001-testing-lisp-test-ob-python.el-Ignore-flaky-tests.patch
Description: Binary data