Thank you for your replies! Currently, I am preparing environment and I am going to prepare proposal on Google Docs till tomorrow. And then I will share it with you for review and maybe additional improvements to it. 9 maj 2014 04:12 "Chris Johns" <chr...@rtems.org> napisał(a):
> On 9/05/2014 1:48 am, Joel Sherrill wrote: > >> >> On 5/7/2014 6:03 PM, Chris Johns wrote: >> >>> On 8/05/2014 8:15 am, Joel Sherrill wrote: >>> >>>> Hopefully Chris Johns will pipe up and add more details. >>>> In particular, where his rtems-test is. I think it can be used >>>> on the beagle board and some gdb simulators now. >>>> >>> The RTEMS Tester is an RTEMS Tools Project and can be found in the >>> rtems-tools.git repo. The web interface to the repo is >>> http://git.rtems.org/rtems-tools/. The tester is under the tester >>> directory. There is also a doc directory with some very basic doco. >>> >>> It supports the gdb run command, gdb via the MI interface with gdb >>> simulators and actual hardware via BDM or JTAG, and qemu is support. It >>> provides serial port access with full termios control. >>> >>> The supported BSPs can be seen here ... >>> >>> http://git.rtems.org/rtems-tools/tree/tester/rtems/testing/bsps >>> >>> The testing back end support can be seen here .. >>> >>> http://git.rtems.org/rtems-tools/tree/tester/rtems/testing >>> >>> On 5/7/2014 4:18 PM, Krzysztof Mięsowicz wrote: >>>> >>>>> Hi, >>>>> >>>>> My name is Krzysztof Mięsowicz. I am interested in participating again >>>>> in SOCIS this year and I am writing to you in hope that you could >>>>> provide me some more details about possible projects related to new >>>>> rtems-test infrastructure. >>>>> >>>>> As dr Joel wrote on Rtems-Users facebook group, there are some >>>>> possible improvements: (cited) >>>>> >>>>> This year we want to encourage the SOCIS students to focus on >>>>> improvements to our new rtems-test Python-based framework. This >>>>> will include: >>>>> + adding more BSPs (simulators and real boards) to rtems-test >>>>> >>>>> Chris will have to point us to his rtems-test repository. I can't >>>> seem to >>>> locate it. >>>> >>> See above. >>> >>> It is in Python and drives real hardware debuggers >>>> and simulators using the GDB/MI. >>>> >>>> rtems-testing/sim-scripts supports more simulators but does not support >>>> hardware debuggers. rtems-test supports a small subset of the simulators >>>> supported by rtems-testing/sim-scripts. rtems-test needs to be worked >>>> on so it can replace all uses of sim-scripts and obsolete them. >>>> >>>> + building more simulators with the RTEMS Source >>>>> - if successful, sim-scripts should be obsolete >>>>> >>>>> That should be RTEMS Source Builder. >>>> http://git.rtems.org/rtems-source-builder/ >>>> RSB builds the basic tools and a couple of simulator configurations. >>>> RSB and rtems-tools work together to provide a framework to manage >>>> build instructions and patch sets for RTEMS tools. We want to expand >>>> it so it can build every tool one needs cross. Simulators are the next >>>> step. >>>> >>> The RSB builds qemu on most hosts. Building qemu for Windows as a >>> Canadian cross is complex and I have not managed to sort this out so >>> Windows is not working. Supporting Windows is outside the scope of this >>> work. >>> >>> There is work in tested the qemu's build and to get BSP support added to >>> the rtems-tester. I am sure as you point out later in this email >>> specific patches are needed to make things work. >>> >>> Eventually, we would like to be able to even build add-on libraries >>>> to target a specific BSP. But that is longer down the road. If someone >>>> gets to it sooner, great! >>>> >>>> I have added some basic support for this via pkgconfig. The RSB >>> contains >>> a private implementation and that can be used to help manage this task. >>> I still have some things to sort out and I am not sure this is ready to >>> have libraries added. >>> >> Agreed. As I say below, we need to focus on automating and improving >> what we have >> now, not growing that. >> >> With that said, it would be nice to see if rtems-addon-packages could be >> killed with >> a few scripts in the RSB. :) >> > > Yes in the end the process needs to be as simple as possible, if however > the RSB needs functionality added I need to get that done first. Here is > the road block. Currently this activity is unfunded so it waits until I > have the time. It is a shame it is taking time because the benefit to users > in having easy to access 3rd party libraries is big. > > RSB needs to be augmented to be able to build all the simulators >>>> external to gdb that we currently use. This may involve some odd >>>> cases like say an old qemu version plus a patch to do coldfire. >>>> That would give a baseline to upgrade against. >>>> >>> Do you have the patches ? The RSB qemu support currently references some >>> zynq patches on Qemu's patchworks so if you can find them there we can >>> directly reference them. >>> >> I would have to dig through them but unfortunately, some are against >> very specific >> versions of qemu and have not been updated. I don't know how we want to >> address >> this. Say coldfire qemu starts with version X + patch as a baseline and >> them push >> that to the newest? >> >> > Can you build qemu with the RSB and try and see what works and what fails > ? They might be fixed. The RSB is building a git version of qemu. > > You know how accepting qemu is of patches. :( >> > > I wish I did know what happens here. It is a complete mystery to me. ;) > > FYI the coldfire patch is Till's and I attempted once in July 2011 to >> submit it. >> I don't recall if I ever followed up on Alexander's comments. I think >> the last >> time I tried, the code had been reworked to the point I couldn't tell >> what to >> do quickly. >> >> http://lists.nongnu.org/archive/html/qemu-devel/2011-07/msg02067.html >> >> Certainly taking another stab at that patch would be a good thing. >> Especially >> since we can do coverage analysis with runs on qemu. I did uC5282 using >> that patched qemu. >> > > The qemu patches are listed here .. > > http://patchwork.ozlabs.org/project/qemu-devel/list/ > > and a cli client for patchworks is here ... > > http://patchwork.ozlabs.org/help/pwclient/ > > With the client you can search for patches. I found it easier than the web > interface. The pwclient config for qemu is a link on this page .. > > http://patchwork.ozlabs.org/project/qemu-devel/ > > >> The arm zynq would also be a good candidate since we could get SMP >> coverage also from a single free simulator. >> > > The zynq is built by the RSB and works with the rtems-tester. > > + integrating coverage runs into rtems-test >>>>> >>>>> rtems-testing/covoar (C++) and rtems-testing/rtems-coverage (shell) >>>> together do RTEMS test runs for coverage and generate the coverage >>>> reports on http://www.rtems.org/ftp/pub/rtems/people/joel/coverage/ >>>> >>>> The rtems-test tool needs to be able to do coverage runs. It needs to >>>> know >>>> if a target supports coverage, how to turn on coverage for a target, how >>>> to run the report generator, etc. >>>> >>>> This code needs to move into the rtems-tools repo under tester/covoar >>> and a waf build script added. This will make the rtems-tools.git repo >>> self contained. >>> >> covoar is not RTEMS specific. The associated shell scripts which drive >> it in an >> RTEMS specific way also need to be included in this. >> >> Chris asked privately about report generation. >> >> The per-coverage configuration/BSP run report generation is completely >> contained >> in covoar (C++). A single run is done by covoar. The wrapper for >> generating multiple >> bsps and the index pages is in the shell script. >> >>> + improvements to the coverage test reporting >>>>> - reporting should be on "module" level. Too coarse now. >>>>> >>>>> The report generator itself needs a refresh. Right now, it reports on >>>> two >>>> "lumps" of code -- core and developmental. Read "critical core" and >>>> "the kitchen sink". :) >>>> >>> When you say report generator do you mean coverage report generator ? >>> >>> What host tools does this report generator use ? My aim is to have >>> rtems-tools as self contains as possible. >>> >> No external tools. Direct output from covoar for the per BSP reports and >> shell script wrapper for the summary tables of runs per BSP/configuration >> and the top level list of BSPs. >> > > Ok. The shell scripts may be converted to Python or the tester config > language. > > Ideally, the coverage reports should be generated based on modules >>>> or source directories in cpukit/. Then we can get the coverage on >>>> score/ separate from shell/, fatfs/, or imfs. What we have seen is that >>>> each time we add a module to the "kitchen sink" set, the covered >>>> percentage drops because the newly added module needs improvement >>>> in test coverage. >>>> >>> Do have more details about this task and how it would proceed ? >>> >> The first thing to understand is that covoar takes a set of .exe's, >> coverage files >> from the simulators, and a list of symbols of interest. You can run covoar >> multiple times against the same set of .exes/.cov files with different >> symbol >> sets. >> > > Ok. I wonder if multiple tables and used and a single pass over the trace > data. It might be faster. This is something we can look at once the basics > are integrated. > > >> The procedure is as follows: >> >> (1) build rtems in a specific configuration. >> Right now, options are -O2 or -Os and POSIX enabled/disabled. >> Eventually, we need to add smp enable/disable for some BSPs. >> >> (2) run all tests on simulator with trace/coverage enabled >> >> (3) wrapper script does: >> for each "subset of source" >> generate a list of symbols from compiled object >> run covoar >> For example, core and developmental (all) reports are two >> invocations >> of covoar with two different symbol lists. >> >> (4) Wrapper shell script, then adds the runs to the per BSP/config page >> and generates the top level BSP index.html page in case it is a >> new BSP. >> >> Right now, each run of covoar is an entry in the "per BSP/config" page. >> I think it will change to where a single coverage test/report run invokes >> covoar multiple times and generates multiple report sets that are >> part of the single "run for a configuration". With each report on a >> symbol set being a subdirectory under the master "this run" directory. >> >> The directory structure will end up being an extension of the current with >> the directory name encoding being tweaked: >> >> + erc32-OsP (run for erc32 at -Os and POSIX enabled) [0] >> index.html - nice links to sub-areas, summation, etc. >> - all/ - equivalent to a "developmental" report now >> - critical/ - equivalent to core now (name can change) >> - score/ - contents look like a current coverage report >> like [1] but focused on a single "module" named score. >> - rtems/ ... >> - posix/ ... >> - libcsupport/ ... >> - fatfs/... >> ... >> - there may be 20 or more of these. Each requires generating a >> symbol set and running covoar. >> >> [0] Note that the last letter is not D or d. This is a type of report >> based on a particular symbol subset being analyzed for this build. >> We may want [Ss] on the end of the name to indicate S for SMP >> enabled or s for disabled. Add s now always until SMP runs >> supported. >> [1] >> http://www.rtems.org/ftp/pub/rtems/people/joel/coverage/ >> erc32/erc32-Ospd-20121220-1418/ >> >> >> >> covoar can generate .gcov output but no one has ever validated it >>>> against "truth" and seen if gcov, lcov, etc. can produce trustworth >>>> reports with it. >>>> >>>>> - some scripts could be migrated to Python >>>>> >>>>> >>>>> The older code is in bash. Migrating it to Python as it is reworked is >>>> desirable. >>>> >>>>> I wonder which of these are of highest priorities and what will it >>>>> mean exactly. Could you provide more details about these ideas and >>>>> maybe point some starting points to get more familiar and prepare >>>>> better proposal? >>>>> >>>>> Everything is high priority. :) >>>> >>>> I guess IMO I would like to see one BSP/simulator pushed through the >>>> entire software stack, reporting improved, and shell transitioned to >>>> Python as it was convenient/expedient. Adding other simulators >>>> to the RSB and rtems-test/ is a bonus. So if you want to assume we >>>> get one student from SOCIS, that would be my suggestion. >>>> >>>> My thinking is that if SOCIS gives us two or more students, one could >>>> easily start at the rtems-test/ level integrating coverage runs as they >>>> are now. Another student could focus on the coverage report generation. >>>> >>>> Transitioning more simulators from sim-scripts/ to rtems-test/ >>>> and adding building more simulators to the RSB could be added >>>> easily. Each simulator instance is more or less independent so >>>> any number of students could work off a check list. >>>> >>>> If we get two or more students, the problem can be tackled from >>>> multiple ends. >>>> >>> Another and more challenging task is to add support to test networking. >>> This would be via the VDE support in qemu. See >>> http://wiki.v2.cs.unibo.it/wiki/index.php/VDE. I do not have anything >>> more to add on this topic other than this looks like a nice to do this. >>> >> >> I would view this as a longer term goal. We do need to test networking but >> we also need improvements in the automation of running, posting, >> interpreting, and generating alerts for the functional, coverage, and >> performance >> tests we have now. >> >> I would rather have more focus on what we have and not add networking as >> a requirement. >> Better to test less well than more in a poor manual fashion. >> > > I agree. I also know the standard from the last GSoC and CGI was so high I > needed more challenging tasks. :) > > Chris > > Regards, >>>>> Krzysztof >>>>> >>>> -- >>>> Joel Sherrill, Ph.D. Director of Research & Development >>>> joel.sherr...@oarcorp.com On-Line Applications Research >>>> Ask me about RTEMS: a free RTOS Huntsville AL 35805 >>>> Support Available (256) 722-9985 >>>> >>>> >>>> >>>> _______________________________________________ >>>> rtems-devel mailing list >>>> rtems-devel@rtems.org >>>> http://www.rtems.org/mailman/listinfo/rtems-devel >>>> >>>> >>
_______________________________________________ rtems-devel mailing list rtems-devel@rtems.org http://www.rtems.org/mailman/listinfo/rtems-devel
