On 03/05/2012 11:40 AM, Paul Larson wrote:
Looping in the linaro-validation mailing list. Nothing private here and would like to get input from others as well.On Mon, Mar 5, 2012 at 11:12 AM, Andy Doan <[email protected] <mailto:[email protected]>> wrote: On 03/02/2012 11:50 PM, Paul Larson wrote: On Thu, Mar 1, 2012 at 4:54 PM, Andy Doan <[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> wrote: This email spans 3 areas and I wasn't sure who'd be interested. Feel free to add people if you'd like. I just spent the day getting the coremark benchmark to run under lava-android-test and wanted to share my findings and methodology. Neat! The key obstacle is that coremark requires a registered download. ie - you can't just throw this in the Android build and let the world use it. This is what makes the whole process trickier than normal "androidify + lavaitize" something. Well, hopefully not too much. A couple of observations here. First off, feel free to tell the LAVA team that you have a tarball you want to host for download into lava, inside the lava infrastructure so that it will be available to LAVA, but not outside. We have a similar situation with the binary drivers needed for panda builds. The scripts that the Android team put together were useful for getting started, but certainly not something we'd want to work with since it required interaction. What we ended up doing was building a tarball of everything it installed, and hosting it inside the lab. We could do a similar thing with coremark so that it's available for lava jobs, but not for public download. We could merge something like this in lava-android-test, but if so, we should include instructions in case anyone outside linaro wants to use it. To use it, they would have to host coremark themselves since we can't provide it. Another option is to do it out of tree, but while lava-test supports out-of-tree tests, it so far hasn't been a priority to push this feature in to lava-android-test since the current push from Zach's team has been to include all their tests. I've expected that there will eventually be some things like this where that's not really possible though, and we have a blueprint to add out-of-tree support to lava-android-test. If this is a model you would need, just let me know and I can bump the priority on that. My current code was done for maximum flexibility. It could be checked into lava-android-test, or we could keep it out-of-tree to avoid confusion for people who'd think they could easily run it. I'm leaning to out of tree, but don't really care. The question I'm thinking about now is about the job submission to LAVA. I'm thinking of an approach where we create a new dispatcher command that would be semi-generic for all restricted tests and look like: { "command": "lava_android_test_install___restricted", "parameters": { "tests": ["coremark"], "install_options": ["coremark_url=http://user:[email protected]/coremark-linaro4.6"__] } }, Why would it need a _restricted version of the command? I don't think that's necessary.
I didn't know if the current command would support the new field I was proposing.
I have one security concern with this approach: Do we enforce any
type of security about who can view the job definitions in LAVA? ie
- if anyone could see the job definition then they'd be able to see
how they could download the restricted file.
I don't think that's a problem. In fact, what I'm suggesting is to
simply have it available as a download from our NAS system in the lab,
that is behind the firewall and not exposed. You'll need to get us to
put the file there, but then it wouldn't need to be behind yet another
layer of security.
The problem is that its not just a single file. I need one file per toolchain we'd like to compare. So for coremark we might have: coremark-android4.4.3, coremark-linaro4.5, coremark-linaro4.6-2012-01, coremark-linaro4.6-2012-02. Additionally, people might also be interested in experimenting with one-off type build configurations. So we need some mechanism to instruct the LAVA test (coremark.py) which binary it should be using.
Given the permutations, and that this will at least change monthly, I thought it would be nice to not depend on having to get some file(s) update in the validation lab.
In fact, as I'm thinking about this, I don't even see a need for an out-of-tree test support in lava-android-test. The out-of-tree support is needed in lava-test because it's getting reinstalled every time. With lava-android-test, it could simply be a python module that we install one time and update as needed, then that test would be available on our server. If anyone else wanted to use it, they could, but would need to specify their own location for getting the coremark binaries.
Correct - I specifically wrote the code this way so that would be a possibility.
Thanks, Paul Larson
# Copyright (c) 2012 Linaro # Author: Linaro Validation Team <[email protected]> # # This file is part of LAVA Android Test. # # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. import os import lava_android_test.config import lava_android_test.testdef test_name = 'coremark' config = lava_android_test.config.get_config() inst_dir = '%s/%s' % (config.installdir_android, test_name) options_file = "%s/install_options" % (inst_dir) coremark = '%s/coremark' % (inst_dir) curdir = os.path.realpath(os.path.dirname(__file__)) script_host = '%s/%s/download_coremark.sh' % (curdir, test_name) script_target = '%s/download_coremark.sh' % (inst_dir) INSTALL_STEPS_ADB_PRE = [ 'push %s %s ' % (script_host, script_target), 'shell chmod 777 %s' % script_target, 'shell %s %s %s' % (script_target, options_file, coremark), ] ADB_SHELL_STEPS = [ '"%s 0x0 0x0 0x66 0 7 1 2000"' % coremark ] PATTERN = "^CoreMark 1.0\s*:\s+(?P<measurement>\d+\.\d+) / (?P<message>.*)" inst = lava_android_test.testdef.AndroidTestInstaller( steps_adb_pre=INSTALL_STEPS_ADB_PRE) run = lava_android_test.testdef.AndroidTestRunner( adbshell_steps=ADB_SHELL_STEPS) parser = lava_android_test.testdef.AndroidTestParser(PATTERN, appendall={'test_case_id':'Iterations/Sec'}) testobj = lava_android_test.testdef.AndroidTest(testname=test_name, installer=inst, runner=run, parser=parser)
download_coremark.sh
Description: Bourne shell script
_______________________________________________ linaro-validation mailing list [email protected] http://lists.linaro.org/mailman/listinfo/linaro-validation
