Hello Zach, On Wed, 28 Mar 2012 12:05:46 -0500 Zach Pfeffer <[email protected]> wrote:
> On 27 March 2012 06:32, Paul Sokolovsky <[email protected]> > wrote: > > Hello, > > > > Yong Qin is working on the blueprint > > https://blueprints.launchpad.net/lava-android-test/+spec/modify-android-build > > to add arbitrary custom client-side scripts to Android Build. He > > submitted first implementation of it as > > https://code.launchpad.net/~liuyq0307/linaro-android-build-tools/run-custom/+merge/98825 > > and documented as > > https://wiki.linaro.org/Platform/Android/AndroidBuild-LavaIntegration . > > > > Unfortunately, I'm not thrilled with that implementation, more > > specifically, its "user interface" (i.e. any parts which user > > directly faces) by the following reasons: > > > > 1. The idea behind Android Build's build config was that they're > > short and easy for human to parse, essentially one glance-over > > would enough to get a good idea what is built here, even for > > outsider. Consequently, the configs should not be overloaded with > > details not related to building. If there's a need for integration > > with other systems, we have good pattern of externalizing such > > details and then just referring to them with a single variable in a > > build config. > > > > 2. The whole approach in > > https://wiki.linaro.org/Platform/Android/AndroidBuild-LavaIntegration > > seems like trying to encode hierarchical structure in the shell > > syntax, which is not much supporting of that. The end result looks > > pretty much like representation of graph structure in raw assembler > > - it's spaghetti mix of data pieces and labels, requiring long time > > to wrap hand around to understand it, and cumbersome and > > error-prone to write. > > > > > > So, I would like to propose alternative syntax solving the issues > > above. I probably should start with saying that if the talk is about > > LAVA, then using native LAVA JSON request immediately comes to mind. > > Well, I guess human-writability wasn't a design goal for that, so I > > skip it. It still makes sense to stick to general-purpose > > hierarchical structure syntax though. Except that JSON has 2 > > problems: a) it doesn't support comments natively, so we'll need to > > pre-process it; b) error reporting/localization may be still no > > ideal. > > > > Anyway, here's my try, it is presented as a comment to > > https://wiki.linaro.org/Platform/Android/AndroidBuild-LavaIntegration > > and then full example at > > https://wiki.linaro.org/Platform/Android/AndroidBuild-LavaIntegration/pfalcon > > > > > > Let's discuss if that covers our needs and constraints. > > Paul, thanks for the alternative proposal. > > The request was: > > Provide a easy way to pass a test into LAVA from an android-build > configuration. > > Yong Qin's approach seems reasonable because its of the form: > > MY_DEFINE="runmycommand --paramone 1 --paramtwo 2" Well, not quite, example at https://wiki.linaro.org/Platform/Android/AndroidBuild-LavaIntegration shows 20 cross-referencing lines for defining tests (with build configuration per se taking 5). > > I don't like Paul's proposal because if you need something that > complicated you should just use the LAVA python interface. I just want > something simple. > > Something like: > > LAVA_TEST_1="runmycommand --paramone 1 --paramtwo 2" > LAVA_TEST_PARSE_1="some regex" > > LAVA_TEST_2="runmyothercommand --paramone 1" > LAVA_TEST_PARSE_2="some other regex" > > ..and that's it. Well, if it's about "parsing", it's never that simple. Yong Qin's example shows what typical parsing regexp looks like. > LAVA would then run LAVA_TEST_1 and LAVA_TEST_2 > etc... and use LAVA_TEST_PARSE_1 and LAVA_TEST_PARSE_2 to parse the > results nad send them back to the android-build. > > All the tests we run on Android will be baked into the build, so we > don't have to worry about installing them. > > I don't even want to have to do this: > > LAVA_TESTS="LAVA_TEST_1 LAVA_TEST_2" > > ...because that's adding too much structure. But if you have too many things without structure, mess ensues. In particular, it's unclear when these tests run in respect to LAVA_TEST_PLAN. > > This is really just a way to get a test in without needing to add a > bunch of python code to LAVA. The flat list of commands is easy to > see, and there's no tree structure to contend with. If you need more > structure you'd write some python or write the structure into a > script. Well, all in all, what you described is quite different from what is presented at https://wiki.linaro.org/Platform/Android/AndroidBuild-LavaIntegration (and implemented in 100 lines (50% of existing code size) at https://code.launchpad.net/~liuyq0307/linaro-android-build-tools/run-custom/+merge/98825) , so probably lack of proper spec to start with was the issue here. If we reduce the requirements to just supporting textual (no external URL references) content in LAVA_TEST_${N} LAVA_TEST_PARSE_${N} and well defined order to run them (for example, after LAVA_TEST_PLAN), then it certainly will be good enough to put into build configs(though parsing regexps would still look hairy there imho). -- Best Regards, Paul Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog _______________________________________________ linaro-validation mailing list [email protected] http://lists.linaro.org/mailman/listinfo/linaro-validation
