Re: [Libreoffice-qa] Extending subsequent tests with dogtail tests?

[Markus Mohrhard]

Hey,

On Fri, Mar 8, 2019 at 6:33 PM Miklos Vajna  wrote:

> Hi,
>
> The current practice is: if 'make check' passes (which is more or less
> enforced by Jenkins) and the change looks good to a reviewer, the change
> goes in. And then releases are based on time, so it's really rare that
> there are "blocker" bugs which would delay a release.
>
> The ideal for any new kind of testing (including accessibility) is that
> it's integrated into 'make check', and whatever those tests cover are
> not OK to be broken anytime.
>
> If the proposed a11y tests are part of make check, then it's easy to
> promise that they won't be broken; otherwise it's just a best effort
> thing without any guarantees.
>
> At least that's how I understood what Markus wrote, and I agree with
> that.
>

Exactly that. With the added comment about the reliability necessary or
developers will start ignoring test failures or even worse disable failing
tests.

Kind regards,
Markus


> Regards,
>
> Miklos
>
___
List Name: Libreoffice-qa mailing list
Mail address: Libreoffice-qa@lists.freedesktop.org
Change settings: https://lists.freedesktop.org/mailman/listinfo/libreoffice-qa
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://lists.freedesktop.org/archives/libreoffice-qa/

Re: [Libreoffice-qa] Extending subsequent tests with dogtail tests?

[Alex ARNAUD]

Le 07/03/2019 à 19:32, Markus Mohrhard a écrit :
> Most likely no. If we have tests for something it is more likely that it
> will be fixed after it is discovered by in the end an accessibility
> regression is similar to any other regression. That means that a test
> started failing you are expected to inspect the test failure before
> commiting but we will most likely not stop a release if a new test
> discovers that something is broken. In general while we do much to avoid
> regressions and having tests helps we don't treat regressions as
> complete release blockers. There is always a case-by-case analysis
> neccessary.

Do you mean someone will break something in LibreOffice and will fix it 
in the future or do you mean someone break accessibility and Hypra has 
to fix it itself in the future ?


In the last scenario, I don't see any reason to have non-regression test 
as we already know that people usually don't take care of accessibility 
and it's why we're forced to keep LibreOffice 4.2 for now. If we expect 
to have one developer fixing bug of thousand of others (on many free 
software) especially because it's accessibility I don't think it's reliable.


Best regards,
Alex.
___
List Name: Libreoffice-qa mailing list
Mail address: Libreoffice-qa@lists.freedesktop.org
Change settings: https://lists.freedesktop.org/mailman/listinfo/libreoffice-qa
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://lists.freedesktop.org/archives/libreoffice-qa/

Re: [Libreoffice-qa] Extending subsequent tests with dogtail tests?

[Markus Mohrhard]

Hey,

let me at least add some comments.

On Thu, Mar 7, 2019 at 5:42 PM Jean-Philippe MENGUAL <
jean-philippe.meng...@libreoffice.org> wrote:

> Hi,
>
>
> Le 06/03/2019 à 20:52, Samuel Thibault a écrit :
> > Hello,
> >
> > Markus Mohrhard, le lun. 25 févr. 2019 00:34:33 +0800, a ecrit:
> >> On a slightly related note I think that we have already quite a few
> tests for
> >> the accessibility UNO layer but as that layer is full of bugs many of
> the tests
> >> are disabled. It might be a good idea to work on these tests before
> actually
> >> trying to implement more complex tests that depend on lower layers
> working
> >> correctly.
> >
> > I'm not sure which piece you are referring to.  Is that the AWB?  I
> > indeed see some source code in toolkit/test/accessibility but no
> > reference to it.
>

There should be accessibility UNO API tests in most top level modules. E.g.
from a quick git grep Accessible -- qadevOOo/Jar_OOoRunner.mk:

qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessible \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleAction \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleComponent \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleComponent \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleContext \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleEditableText \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleEventBroadcaster \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleExtendedComponent \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleImage \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleSelection \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleTable \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleText \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/ifc/accessibility/_XAccessibleValue \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_basctl/AccessibleShape \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/AccessibleEditableTextPara_HeaderFooter \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/AccessibleEditableTextPara_PreviewCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sch/AccessibleDocumentView \
qadevOOo/Jar_OOoRunner.mk:qadevOOo/tests/java/mod/_sc/ScAccessibleCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleCsvCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleCsvGrid \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleCsvRuler \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleDocument \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleDocumentPagePreview \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessiblePageHeader \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessiblePageHeaderArea \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessiblePreviewCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessiblePreviewHeaderCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessiblePreviewTable \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sc/ScAccessibleSpreadsheet \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sd/AccessibleDrawDocumentView \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sd/AccessibleOutlineView \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sd/AccessibleSlideView \
qadevOOo/Jar_OOoRunner.mk:qadevOOo/tests/java/mod/_sm/SmEditAccessible \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_sm/SmGraphicAccessible \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBox \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBoxHeaderBar \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBoxHeaderCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBoxTable \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleBrowseBoxTableCell \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleIconChoiceCtrl \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleIconChoiceCtrlEntry \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleTabBar \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleTabBarPage \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleTabBarPageList \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleTreeListBox \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svtools/AccessibleTreeListBoxEntry \
qadevOOo/Jar_OOoRunner.mk:
qadevOOo/tests/java/mod/_svx/AccessibleControlShape \
qadevOOo/Jar_OOoRunner.mk:

Re: [Libreoffice-qa] Extending subsequent tests with dogtail tests?

[Jean-Philippe MENGUAL]

Hi,


Le 06/03/2019 à 20:52, Samuel Thibault a écrit :

Hello,

Markus Mohrhard, le lun. 25 févr. 2019 00:34:33 +0800, a ecrit:

On a slightly related note I think that we have already quite a few tests for
the accessibility UNO layer but as that layer is full of bugs many of the tests
are disabled. It might be a good idea to work on these tests before actually
trying to implement more complex tests that depend on lower layers working
correctly.


I'm not sure which piece you are referring to.  Is that the AWB?  I
indeed see some source code in toolkit/test/accessibility but no
reference to it.


The focus handling can be easily integrated into the existing UI
testing infrastructure and might benefit there from some of the concepts that
should make them more stable


Good :) So could you plan to work on it?


How could such plan be scheduled in LO qa? Should we report a bug? Or 
open a wiki roadmap?


The problem now is that it is not possible to fix accessibility bugs, 
fix regressions from 4.2, integrating in the code non-regression tests, 
and do the same for the three major programs in free software. It is 
less a will problem than a resource problem, because even with funds, we 
do not have enough persons to work on this with skills related to 
Libreoffice and accessibility in general. As you know more and more 
persons go the web or backend techno, less in programming for desktop 
software.


So while we are ready to fix accessibility bugs, we need non-regression 
tests. And it is difficult to do both. The tool on which we are working, 
dogtail, is interesting because enables to test via the same framework 
different programs, without needing to know the code of each of one. I 
think gateway is possible between Libreoffice framework test and such 
tool. We also could imagine a dedicated machine with dogtail to test, 
but should be acceptd. Also, the thing is to know if LO is ready to 
prevent a release or a commit because introduces a regression in 
accessibility.


Well to sum up, beyond our effort, that we try general and 
cross-software, we would need to know how we can together set a kind of 
roadmap to implement accessibility in the existing frameworks, and add a 
layer to make common scenarios between our general tool and LO's one. 
Having that may be in the easy hacks to have contributions? Or b a 
specific TDF tender just like we did for labels?


Regards


Samuel
___
LibreOffice mailing list
libreoff...@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/libreoffice



--
Logo Hypra  JEAN-PHILIPPE MENGUAL
DIRECTEUR TECHNIQUE ET QUALITÉ
102, rue des poissonniers, 75018, Paris
Tel : +331 84 73 06 61  Mob : +336 76 34 93 37

jpmeng...@hypra.fr 
www.hypra.fr 
Facebook Hypra  Twitter Hypra
 Linkedin Jean-Philippe



___
List Name: Libreoffice-qa mailing list
Mail address: Libreoffice-qa@lists.freedesktop.org
Change settings: https://lists.freedesktop.org/mailman/listinfo/libreoffice-qa
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://lists.freedesktop.org/archives/libreoffice-qa/

Re: [Libreoffice-qa] Extending subsequent tests with dogtail tests?

[Noel Grandin]

On Sun, 24 Feb 2019 at 22:04, Samuel Thibault  wrote:

> The fact that even the C UI strings may change is a concern indeed. We
> however don't really have another way to identify widgets, do we? (we
> don't want to identify them structurally, that'd be even less stable)
>
>
The LibreOffice python UI tests use IDs that are maintained in the .ui
files (and are just strings), which are relatively stable.
___
List Name: Libreoffice-qa mailing list
Mail address: Libreoffice-qa@lists.freedesktop.org
Change settings: https://lists.freedesktop.org/mailman/listinfo/libreoffice-qa
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://lists.freedesktop.org/archives/libreoffice-qa/

Re: [Libreoffice-qa] Extending subsequent tests with dogtail tests?

[Markus Mohrhard]

Hello Samuel,

let me add a few comments based on many years implementing testing
frameworks for LibreOffice and most recently working on the UI testing.

On Sat, Feb 23, 2019 at 6:24 PM Samuel Thibault  wrote:

> Noel Grandin, le sam. 23 févr. 2019 12:19:19 +0200, a ecrit:
> > However, the current python UI test stuff talks directly to the vcl/
> widgets.
> > But between the vcl/ widgets and an actual accessibility user lies at
> least two
> > major chunks of code - the generic accessibility/ stuff, and the
> operating
> > system specific bridges.
>
> Yes, that's my concern with testing only at the vcl layer.
>
> That said, we could as well make tests work at both layers. Run them
> along uitests, thus very frequently, and run them periodically through
> the accessibility layer on a system which has it.
>
>
If my understanding of your idea is correct you want to test actually two
things that are required for working accessibility for users. On one hand
you need to test the accessibility APIs and on the other hand correct focus
handling. I think you most likely want to test them independently if
possible as I think you'll discover that combining them will lead to hard
to diagnose problems. Additionally, the target of any test framework has to
be that it minimizes the number of random test failures and requires as
little adoption for unrelated code changes as possible.

I noticed after quickly looking at your proposed tests is that you use UI
strings in your tests which I think you want to avoid as much as possible.
There are several problems with UI strings that make them a bad property
during testing: they change often, often not even by developers and are
localized. Especially the second point means that your tests suddenly only
work in en-US which surely limits a bit the usefulness of the tests. Even
more if you plan to generate test cases automatically as it means that the
tests can only be generated with an en-US locale.

On a slightly related note I think that we have already quite a few tests
for the accessibility UNO layer but as that layer is full of bugs many of
the tests are disabled. It might be a good idea to work on these tests
before actually trying to implement more complex tests that depend on lower
layers working correctly. The focus handling can be easily integrated into
the existing UI testing infrastructure and might benefit there from some of
the concepts that should make them more stable (deadlock detection,
addressing UI elements through ID instead of UI visible strings, mostly
working async dialog and action handling) and I think it makes sense to
check how much of the accessibility part can not be covered with a
combination of UI testing and accessibility UNO API testing. It might even
be possible to get accessibility event interception into the UI testing to
check that events are generated correctly during actions. I would need to
look into how our accessibility handling works in the VCL layer to be sure
that this actually works.

For the dependencies we surely don't want to include the source code itself
into LibreOffice's external dependency handling. At least for the beginning
the easiest approach might be something like an
--enable-accessibility-tests flag which can be checked in configure.ac
where we make sure all required dependencies are around. If we ever decide
that they should be run by default we can still switch the default value
and make it possible for people to override the setting in their autogen
settings.

On a slightly related note, please make sure that you are developing
against python 3. I did not check if you are actually using any pure
python2 features but at least "#!/usr/bin/python" will get you a python 2
interpreter on most systems.

Kind regards,
Markus
___
List Name: Libreoffice-qa mailing list
Mail address: Libreoffice-qa@lists.freedesktop.org
Change settings: https://lists.freedesktop.org/mailman/listinfo/libreoffice-qa
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://lists.freedesktop.org/archives/libreoffice-qa/

Re: [Libreoffice-qa] Extending subsequent tests with dogtail tests?

[Noel Grandin]

So yes, the focus information would be fairly easy to expose to the python
UI test infrastructure.

And yes, it would be good to limit the number of different frameworks we
use for testing (we already have 4 - C++ direct, C++ via UNO, Java via UNO,
python via UNO)

However, the current python UI test stuff talks directly to the vcl/
widgets. But between the vcl/ widgets and an actual accessibility user lies
at least two major chunks of code - the generic accessibility/ stuff, and
the operating system specific bridges.

And it would be good to test those too, which this dogtail-based stuff
would do.

 just laying out the pros and cons... :-)
___
List Name: Libreoffice-qa mailing list
Mail address: Libreoffice-qa@lists.freedesktop.org
Change settings: https://lists.freedesktop.org/mailman/listinfo/libreoffice-qa
Problems? http://www.libreoffice.org/get-help/mailing-lists/how-to-unsubscribe/
Posting guidelines + more: http://wiki.documentfoundation.org/Netiquette
List archive: http://lists.freedesktop.org/archives/libreoffice-qa/