Hi Lada!
Thanks for the detailed answers, it looks very promising to me. I will
try to do some practice exercise with it to get better understanding of
how it works.
On 10/5/18 12:43 PM, Ladislav Slezak wrote:
Hello!
Dne 04. 10. 18 v 9:48 Rodion Iafarov napsal(a):
I'll comment the requirements from the libyui REST API prototype POV so we could
see if that would fit and we could compare it with the other possibilities:
These is combined list for the needs for your previous email:
1. Operating on all existing widgets, including tables, trees, comboboxes, tabs.
The prototype does not support the complete set of widgets yet (ENOTIME) but
adding support for the missing ones is quite easy. The most complex will be the
package manager widget as from the YaST POV the whole dialog is a single widget
like
a single push button. And they are implemented in separate packages
(libyui-*-pkg).
Being able to perform same set of actions like user can do normally.
It turned out that the prototype was actually more powerful than the user at the
beginning, e.g. I could change the state of a disabled check box, but user
cannot
click it in reality. Later I added a check and disallowed changing the disabled
widgets.
But I think this should not be a problem, we just need to think about it,
carefully
compare the real UI and REST API possibilities and make it as close as possible.
Josef already mentioned that it's important to check that all widgets are
properly
displayed etc... The REST API currently does not provide any information about
the
rendered widgets on the screen.
But it should be possible to get some data from the layout engine or the UI
itself.
Then we could return additional flags like "out-of-screen" or
"partially-displayed"
for each widget and test in the frontend that no widget has these flags set.
2. Possibility to assess widgets properties e.g. values in the table cells,
labels
text, UI properties (if control is visible/disabled)
The REST API works directly on the libyui level, we can read (almost)
everything.
We can get much more details than it is possible from the YaST code.
3. Some way of integration with openQA (that's important for QA team, as we have
to use it, but it should not be an issue as we can simply create file with
results and parse it)
The integration should be simple, you just need to add some REST API client
which can be implemented in any language and process the JSON response.
The problem would be when testing a network-less scenario (e.g. network not
connected
or even no network card present) because an HTTP REST API naturally requires a
network.
But in that case you could connect to the REST API locally via serial console,
run "curl http://localhost/..."; there and pass the output further. That looks a
bit
annoying but should be solvable quite easy with some wrapper.
4. There is not to complex way to launch tests when running installation (e.g.
booting from installer DVD and triggering test)
The prototype is already able to test the installation, you just need to set
some
environment variables via the boot command line to enable the REST API server
(disabled by default) and that's it. Then the installation can be driven from
outside
via the REST API [1].
5. There is a way to run tests with local changes without configuring any kind
of
complex setup
It mainly depends on how you design the frontend part of the testing framework,
the
backend part is trivial and once we integrate it into YaST it will run out of
box (or
you just install few packages before using it).
For example the Cucumber prototype is very easy to use, you only need Ruby and
Bundler installed. Install the gems [2], prepare the environment (start a VM if
needed) and you can run the tests [3].
Right, but not to double the efforts for the test development, we need
some common solution which is then executed on different stages of the
development process. Therefore, it will be beneficial to agree on the
front-end design to use same solution and avoid development of the same
tests.
6. Solution can be used to run CI per branch
I'm not sure what does "branch" mean in this case. Product branch like GA, SP1,
SP2?
Could you clarify?
As per one of our discussions we had during YaST workshop, we agreed
that we want to test earlier. Ideally, we could apply such CI to run
against feature branches, so developers can test their changes even
before creating pull request.
Otherwise, we still could implement CI to run against master + product
branches, or at least just against master. So it's related to point 5 a
bit, but is still different, as we still could use parts of openQA to
spawn VMs, etc.
7. Can be used for ncurses and qt GUIs
The prototype already supports both.
8. Running tests minimizes influence on SUT (we should keep difference of real
installation vs integration one as little as possible not installing hundreds of
the dependencies).
So far it just needs two additional packages (libjsoncpp and libmicrohttpd),
in total it's ~170KB (packed RPMs) so the footprint is really minimal. That's
even
more important for the installer as the inst-sys space is very limited.
In installed system you might additionally need to open the HTTP port at
firewall (or
you can use the same trick as with the network-less installation).
Summary of the most important features:
- The REST API can be used from any programming language, it's language
independent
- It makes a clean boundary between the frontend and the backend
- The internal JSON format is human readable, we could store the responses for
later
debugging if needed
- The frontend may use any testing framework (Cucumber, RSpec, ... or a custom
one),
it can be easily updated/changed
- Small dependencies for the backend
- Works with both Qt and ncurses
[1] https://github.com/lslezak/cucumber-yast#testing-the-yast-installer
[2] https://github.com/lslezak/cucumber-yast#installing-cucumber-ruby-gem
[3] https://github.com/lslezak/cucumber-yast#running-the-tests
--
Kind regards,
Rodion Iafarov <[email protected]>
QA Engineer
SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nürnberg
Tel: +49-911-74053-0; Fax: +49-911-7417755; https://www.suse.com/
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard,
Graham Norton, HRB 21284 (AG Nürnberg)
--
To unsubscribe, e-mail: [email protected]
To contact the owner, e-mail: [email protected]
