Re: [Development] How to run QtBase autotests on a remote machine?

[Stottlemyer, Brett (B.S.)]

On 4/12/18, 2:07 AM, "Development on behalf of Maurice Kalinowski" 
 wrote:

> > > Some tests shall need more than the binary; e.g. data files to act on.
> > 
> [Kalinowski Maurice] 
> Check for testdata or builtin_testdata, I do not recall the precise name 
> right now.
> 
> Basically you can specify whether you want the data to be copied or use qrc 
> to integrate it to the app. The test system then automatically extracts it on 
> the target hardware during execution and uses it.



> Oliver is currently working on getting UWP into it, it shares a lot of 
> concepts with other remote targets. So when we started on it there were a 
> couple of infrastructural things to clarify (like builtin_testdata from 
> above).
> From what I know the other platforms might come at a later point, but should 
> be easier to do, as the technical building blocks are available.

I tested out a different approach, although it is not clear if the above work 
will end up making it obsolete. I added another subproject to QtCreator to 
build a headless application/tool that could leverage the code in QtC to parse 
.pro files and run ssh/scp functions. Given a source and build directory, it 
could detect the files that should be copied to target move them, then 
optionally run additional commands over ssh. The idea being that if QtC is 
smart enough to deploy from the IDE, make the same functionality accessible 
from the command line.

Unfortunately, QtC doesn’t not have the same binary compatibility constraints 
of Qt libs, so it seemed to need updates for every new QtC version. IIRC, 
having it parse the project file with `QT_BUILD_PARTS+=tests` would include the 
test executables and dependencies. The idea was to have some that ran from the 
command line and could be integrated with CI, although we haven’t had time to 
do that integration. It had code paths for Qt modules as well as end-user 
projects.

If there is interest in making this part of QtC itself, I could push the code 
(built against QtC 4.4.1) to gerrit for review.

Regards,
Brett

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] How to run QtBase autotests on a remote machine?

[Maurice Kalinowski]

Hi,
>  >> Currently I build the tests and then scp individual tst_foo binaries  >> 
> to the
> host and run them manually. In order to fix the initial porting  >> problems 
> that
> will show up in most tests this is fine.
>  >
>  > Some tests shall need more than the binary; e.g. data files to act on.
> 
[Kalinowski Maurice] 
Check for testdata or builtin_testdata, I do not recall the precise name right 
now.

Basically you can specify whether you want the data to be copied or use qrc to 
integrate it to the app. The test system then automatically extracts it on the 
target hardware during execution and uses it.

>  >
> 
> When it comes to CI, tests don't seem to be run on Android and iOS targets.
> There are two cross-compilation targets (Linux arm64 and armv7) that are
> tested, but those are under QEMU emulation, not on real hardware.
[Kalinowski Maurice] 
Correct,

Oliver is currently working on getting UWP into it, it shares a lot of concepts 
with other remote targets. So when we started on it there were a couple of 
infrastructural things to clarify (like builtin_testdata from above).
>From what I know the other platforms might come at a later point, but should 
>be easier to do, as the technical building blocks are available.

Maurice 

___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] How to run QtBase autotests on a remote machine?

[Kari Oikarinen]

On 11.04.2018 15:06, Edward Welbourne wrote:
> Alexander Richardson (11 April 2018 10:13)
>> I am currently trying to get Qt to work on the CHERI CPU
>> (http://www.cl.cam.ac.uk/research/security/ctsrd/cheri/cheri-faq.html)
>
> Sounds interesting.
>
>> I have now managed to get the first few unit tests to build and run as
>> CHERI pure capability programs
>
> Impressive :^>
>
>> so I would like to run the full test suite.
>
> Makes sense.
>
>> Currently I build the tests and then scp individual tst_foo binaries
>> to the host and run them manually. In order to fix the initial porting
>> problems that will show up in most tests this is fine.
>
> Some tests shall need more than the binary; e.g. data files to act on.

Unfortunately I don't know of a better way. I asked people who have been 
working

on the INTEGRITY port and they've also just made static builds of test
executables and then copied them over to the device to be run.

>
>> Is there any existing mechanism to run the unit tests on a machine
>> other than the compilation host?
>
> I hope so - does anyone know how we test on Android, iOS or any other
> cross-compilation target ?  Or do we even support testing on them ?
>

When it comes to CI, tests don't seem to be run on Android and iOS targets.
There are two cross-compilation targets (Linux arm64 and armv7) that are 
tested,

but those are under QEMU emulation, not on real hardware.

For QEMU tests the tests are run in the folder they were built in under
emulation, so they don't offer any hints on a good process for testing 
in a real

device.

--
Kari
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] How to run QtBase autotests on a remote machine?

[Joerg Bornemann]

On 04/11/2018 02:06 PM, Edward Welbourne wrote:


That sounds like a promising line of enquiry - except that I can't find
where we generate target_wrapper.sh - the word target_wrapper doesn't
appear to exist in our source tree (aside from a couple of .gitignore
files).  


The wrapper script is generated in 
qtbase/mkspecs/features/qt_functions.prf by qtAddToolEnv. The 
qtPrepareTool calls in our project files trigger the wrapper generation.



Cheers,

Joerg
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] How to run QtBase autotests on a remote machine?

[Tor Arne Vestbø]

> 
> I cannot run the compiler on the target platform due to memory and CPU
> speed constraints so I need to cross-compile everything [3].
> Is there any existing mechanism to run the unit tests on a machine
> other than the compilation host?

Mount the tests on the device (sshfs e.g.), and run the tests there.

Or, if you want to be fancy, on the host, export TESTRUNNER=wrapper.sh, where 
wrapper.sh does something like ssh device $*

Tor Arne 
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

Re: [Development] How to run QtBase autotests on a remote machine?

[Edward Welbourne]

Alexander Richardson (11 April 2018 10:13)
> I am currently trying to get Qt to work on the CHERI CPU
> (http://www.cl.cam.ac.uk/research/security/ctsrd/cheri/cheri-faq.html)

Sounds interesting.

> I have now managed to get the first few unit tests to build and run as
> CHERI pure capability programs

Impressive :^>

> so I would like to run the full test suite.

Makes sense.

> Currently I build the tests and then scp individual tst_foo binaries
> to the host and run them manually. In order to fix the initial porting
> problems that will show up in most tests this is fine.

Some tests shall need more than the binary; e.g. data files to act on.

> Is there any existing mechanism to run the unit tests on a machine
> other than the compilation host?

I hope so - does anyone know how we test on Android, iOS or any other
cross-compilation target ?  Or do we even support testing on them ?

> I have noticed that there is a target_wrapper.sh script in every test
> directory. Can that be modified to execute scp and ssh to run the test
> remotely?

That sounds like a promising line of enquiry - except that I can't find
where we generate target_wrapper.sh - the word target_wrapper doesn't
appear to exist in our source tree (aside from a couple of .gitignore
files).  If you can find what creates these scripts, hacking it would be
one way to go.  However, finding out how we test on other
cross-compilation platforms may save you a whole lot of work ...

Eddy.
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development

[Development] How to run QtBase autotests on a remote machine?

[Alexander Richardson]

Hello,

I am currently trying to get Qt to work on the CHERI CPU
(http://www.cl.cam.ac.uk/research/security/ctsrd/cheri/cheri-faq.html)
running a modified version of FreeBSD.
CHERI is a capability architecture with 128-bit pointers (memory
capabilities) and unlike on most other CPU architectures pointers are
*not* the same as a integers [1]. Compared to other programs and
libraries that I have modified to run on CHERI QtCore has been quite
straightforward [2]. I was pleasantly surprised by how easy it was to
get the configure
checks to pass and add the required custom command line flags to the
build systems.

I have now managed to get the first few unit tests to build and run as
CHERI pure capability programs so I would like to run the full test
suite.
Currently I build the tests and then scp individual tst_foo binaries
to the host and run them manually. In order to fix the initial porting
problems that will show up in most tests this is fine.
However, now I would like to run the full test suite to find any other
issues in Qt and/or our compiler/OS libraries/kernel.

I cannot run the compiler on the target platform due to memory and CPU
speed constraints so I need to cross-compile everything [3].
Is there any existing mechanism to run the unit tests on a machine
other than the compilation host?

I have noticed that there is a target_wrapper.sh script in every test
directory. Can that be modified to execute scp and ssh to run the test
remotely?
Currently, I am only attempting to run statically linked tests, but
once I have managed that I would also like to run them dynamically
linked which will probably require copying more files and setting
QT_PLUGIN_PATH.

Thanks,
Alex



[1] In CHERI we use capabilities instead of pointers. Our capabilities
hold a virtual address as well as metadata describing the memory
resources referenced by the pointer (bounds, permissions, ...) and
also a 1-bit tag that protects the pointer itself (integrity, valid
provenance, ...). This tag bit is cleared when a capability is
overwritten with plain data so there is no way integers and pointers
can be confused.

[2] The only real incompatibility I have found so far is that
QArrayData::data() stores an offset to the data instead of a pointer.
On CHERI pointers can only be created by subsetting an existing
capability.
For the common case where the string data is part of the same
allocation as the QArrayData header using  `return (char*)this +
offset` also works since it is just setting an offset in the allocated
memory region. However, the fromRawData() case does not work since
that requires creating a pointer that is out of bounds of the
QArrayData allocation and will cause a trap when accessed.
Other than this the majority of the work so far was changes to our
compiler+linker and the OS libraries. We have so far focussed mostly
on software written in C (and the libc++ testsuite) so compiling Qt
has exposed quite a few compiler crashes since it is the first big C++
project we have attempted to port.

[3] We can currently run CHERI code on QEMU
(https://github.com/CTSRD-CHERI/qemu) or on an FPGA running at about
100MHz. I could theoretically attempt to run a MIPS version of clang
in QEMU but it is so slow that using that approach is not feasible.
___
Development mailing list
Development@qt-project.org
http://lists.qt-project.org/mailman/listinfo/development