Re: [oe] Alpha release of check-requirement.sh

[Behan Webster]

> On Aug 10, 2017, at 7:26 AM, Leonardo Sandoval 
>  wrote:
> 
> On Wed, 2017-08-09 at 21:40 -0700, Behan Webster wrote:
>>> On Aug 9, 2017, at 7:22 AM, Leonardo Sandoval 
>>>  wrote:
>>> 
>>> in my opinion, this fits nicely into the oe-core, into the
>>> scripts/contrib folder.
>> 
>> Keeping it in script/contrib seems odd to me (horse before the cart). One of 
>> the requirements for OE is git (or tar) to actually get OE in the first 
>> place.
>> 
>> I see this as more something some one would run before cloning OE or Poky. 
>> It’s a replacement for needing to read a long HOWTO on a webpage to get 
>> started with OE or YP.
> 
> 
> this installation way can help
> 
> $ curl  | sh

Well, it would have to be bash (not sh), but that already works. I’m just not 
happy with that way of doing things. Piping arbitrary commands into bash has 
always been a terrible practice IMHO.

Again, remember there is more than one grouping of requirements, so parameters 
need to be passed in which is problematic with that style of running a script.

One of the reasons I have an —install option is so people have more control 
over what is done to their machine. I have people download it and run it 
because quite often people need to fix things and then rerun the script.

Behan
-- 
___
Openembedded-devel mailing list
Openembedded-devel@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

Re: [oe] Alpha release of check-requirement.sh

[Behan Webster]

> On Aug 10, 2017, at 9:49 AM, Behan Webster  wrote:
> 
> There is already a test system for packages per distro. It uses a second 
> script called pkgsearch which uses meta data from pkgs.org  
> (pkgsearch is also in my git repo). You can see part of the test machinery in 
> the —check-packages option in the check-requirements.sh script. (I couldn’t 
> maintain the lists of packages for all the LF courses without an automated 
> test system).
> 


If anyone is interested, pkgsearch is here, though not advertised anywhere (I 
suppose it is now). ;)

https://training.linuxfoundation.org/cm/prep/pkgsearch 


Again, all in git. Just tied into a lot of other LF things right now I need to 
disentangle.

Behan
-- 
___
Openembedded-devel mailing list
Openembedded-devel@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

Re: [oe] Alpha release of check-requirement.sh

[Behan Webster]

> On Aug 10, 2017, at 1:42 AM, Tim Orling  wrote:
> 
> Also, I recommend a method of linking the checker script with the 
> documentation.

That’s actually where this all started. I generate documentation from the JSON 
from my script for he LF training documentation. I’ve been reading the 
documentation to update my package lists, but found a lot of errors in the docs 
(missing packages, typos, duplicate packages, inconsistencies) all which I 
reported.

Since those errors are trivial for me to find because of my test system (more 
below) I’m suggesting the package lists in the documentation be generated from 
my tool (mirroring what I do for ready-for.sh)

Not that I’m suggesting the same for YP, but for LF, I maintain all 
requirements (machine and packages) in ready-for.sh, which generates JSON which 
is used by lualatex to generate per class setup documentation (55 auto 
generated PDFs).

I literally update one file, type make and it generates script, website, and 
PDFs.

I’m suggesting something similar can be done for the YP documentation, but only 
for the package lists.

> And run unit tests to prove nothing has changed on distro X, Y or Z.

There is already a test system for packages per distro. It uses a second script 
called pkgsearch which uses meta data from pkgs.org  
(pkgsearch is also in my git repo). You can see part of the test machinery in 
the —check-packages option in the check-requirements.sh script. (I couldn’t 
maintain the lists of packages for all the LF courses without an automated test 
system).

This is what a successful test run looks like.

$ ./check-requirements.sh --check-packages --all
YP .*..*.**...*...
YPDOC .***.*.*...
YPGUI .**..*.**.*...**..*
YPST .*..*..*..*...*

dots represent a passed package list for a distro. The star represents a 
download of a cross-bistro package list from pkgs.org  (lists 
are cached so they’re only downloaded once per day).

Here’s what a failed test looks like when asking for the "pony” package:

behanw@turin-vm:/mnt/src/lf/github/ready-for/check-requirements$ 
./check-requirements.sh --check-packages --all --nocache
YP .*
NOTE: Checking CentOS-6 for YP...
MISSING: pony (CentOS-6)
.
NOTE: Checking CentOS-7 for YP...
MISSING: pony (CentOS-7)
.
NOTE: Checking Debian-7 for YP...
MISSING: pony (Debian-7)
.
NOTE: Checking Debian-8 for YP...
MISSING: pony (Debian-8)
.


> And I want a pony, with wings and a single horn.

Tim: Is pink okay?

https://thumbs.dreamstime.com/z/pony-unicorn-golden-wings-big-eyes-cloud-rainbow-multicolor-vector-illustration-86109664.jpg
 


Behan ;)
-- 
___
Openembedded-devel mailing list
Openembedded-devel@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

Re: [oe] Alpha release of check-requirement.sh

[Behan Webster]

It is in a git repo. The same git repo as my other tool. It's just entangled in 
a number of other things right now that I need to disentangle. There is 5 years 
of history in building this in that repo. Also 5 years of seeing what works in 
precisely the use case this solves: checking the requirements of a developer's 
computer and making sure the right packages are installed so they can proceed.

The reason it is written in bash, and not python, is because python is not 
universally available on all distros out of the box. This is why python is 
listed in the package list for every distro in the setup documentation. 
Remember this isn't about choose your favourite language, it's about using 
bootstrapping a build environment.

There is literally nothing you need to do other than download it and run it on 
the distros most people use.

I wrote it in bash (and related utilities) because bash is always there. And 
considering ready-for.sh (what check-requirements.sh is based on) has 
successfully been run (minimally) thousands of times successfully (probably a 
lot higher since it is also used for the LF MOOC) I stand by this decision.

And again this is not intended to be an OE or YP only thing. Again this is 
being maintained alongside an existing tool and shares more than 90% of the 
code with ready-for.sh. Moving or rewriting it in $LANGUAGE means essentially 
forking it.

Behan

Sent from my Mobile Computer which is also a phone

> On Aug 10, 2017, at 1:38 AM, Tim Orling  wrote:
> 
> So where does it live? It ends up being: run shell script from "somewhere". I 
> know what you mean about git and other requirements, but still it must be 
> sourced from "somewhere". If you add a desire to be version controlled, it 
> then logically lands in a git repo. Or subversion, bazaar or mercurial.
> 
> Thinking outside the box, we could make it a python (humor me) module 
> installed from pypi and document how to run it...
> 
> I tend to use shell scripts as my first draft and then translate to python. 
> This usually means a lot of "subprocess" calls and is by no means a silver 
> bullet.
> 
> Alternatively, it becomes a clickable link on a website. That means nodejs. 
> Or perl. Or...
>> On Wed, Aug 9, 2017 at 9:40 PM Behan Webster  
>> wrote:
>> > On Aug 9, 2017, at 7:22 AM, Leonardo Sandoval 
>> >  wrote:
>> >
>> > in my opinion, this fits nicely into the oe-core, into the
>> > scripts/contrib folder.
>> 
>> Keeping it in script/contrib seems odd to me (horse before the cart). One of 
>> the requirements for OE is git (or tar) to actually get OE in the first 
>> place.
>> 
>> I see this as more something some one would run before cloning OE or Poky. 
>> It’s a replacement for needing to read a long HOWTO on a webpage to get 
>> started with OE or YP.
>> 
>> Behan
>> --
>> ___
>> Openembedded-devel mailing list
>> Openembedded-devel@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
-- 
___
Openembedded-devel mailing list
Openembedded-devel@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

Re: [oe] Alpha release of check-requirement.sh

[Leonardo Sandoval]

On Wed, 2017-08-09 at 21:40 -0700, Behan Webster wrote:
> > On Aug 9, 2017, at 7:22 AM, Leonardo Sandoval 
> >  wrote:
> > 
> > in my opinion, this fits nicely into the oe-core, into the
> > scripts/contrib folder.
> 
> Keeping it in script/contrib seems odd to me (horse before the cart). One of 
> the requirements for OE is git (or tar) to actually get OE in the first place.
> 
> I see this as more something some one would run before cloning OE or Poky. 
> It’s a replacement for needing to read a long HOWTO on a webpage to get 
> started with OE or YP.


this installation way can help

$ curl  | sh



> 
> Behan


-- 
___
Openembedded-devel mailing list
Openembedded-devel@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

Re: [oe] Alpha release of check-requirement.sh

[Tim Orling]

Also, I recommend a method of linking the checker script with the
documentation. And run unit tests to prove nothing has changed on distro X,
Y or Z. And I want a pony, with wings and a single horn.

On Thu, Aug 10, 2017 at 1:38 AM Tim Orling  wrote:

> So where does it live? It ends up being: run shell script from
> "somewhere". I know what you mean about git and other requirements, but
> still it must be sourced from "somewhere". If you add a desire to be
> version controlled, it then logically lands in a git repo. Or subversion,
> bazaar or mercurial.
>
> Thinking outside the box, we could make it a python (humor me) module
> installed from pypi and document how to run it...
>
> I tend to use shell scripts as my first draft and then translate to
> python. This usually means a lot of "subprocess" calls and is by no means a
> silver bullet.
>
> Alternatively, it becomes a clickable link on a website. That means
> nodejs. Or perl. Or...
> On Wed, Aug 9, 2017 at 9:40 PM Behan Webster 
> wrote:
>
>> > On Aug 9, 2017, at 7:22 AM, Leonardo Sandoval <
>> leonardo.sandoval.gonza...@linux.intel.com> wrote:
>> >
>> > in my opinion, this fits nicely into the oe-core, into the
>> > scripts/contrib folder.
>>
>> Keeping it in script/contrib seems odd to me (horse before the cart). One
>> of the requirements for OE is git (or tar) to actually get OE in the first
>> place.
>>
>> I see this as more something some one would run before cloning OE or
>> Poky. It’s a replacement for needing to read a long HOWTO on a webpage to
>> get started with OE or YP.
>>
>> Behan
>> --
>> ___
>> Openembedded-devel mailing list
>> Openembedded-devel@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
>>
>
-- 
___
Openembedded-devel mailing list
Openembedded-devel@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

Re: [oe] Alpha release of check-requirement.sh

[Tim Orling]

So where does it live? It ends up being: run shell script from "somewhere".
I know what you mean about git and other requirements, but still it must be
sourced from "somewhere". If you add a desire to be version controlled, it
then logically lands in a git repo. Or subversion, bazaar or mercurial.

Thinking outside the box, we could make it a python (humor me) module
installed from pypi and document how to run it...

I tend to use shell scripts as my first draft and then translate to python.
This usually means a lot of "subprocess" calls and is by no means a silver
bullet.

Alternatively, it becomes a clickable link on a website. That means nodejs.
Or perl. Or...
On Wed, Aug 9, 2017 at 9:40 PM Behan Webster 
wrote:

> > On Aug 9, 2017, at 7:22 AM, Leonardo Sandoval <
> leonardo.sandoval.gonza...@linux.intel.com> wrote:
> >
> > in my opinion, this fits nicely into the oe-core, into the
> > scripts/contrib folder.
>
> Keeping it in script/contrib seems odd to me (horse before the cart). One
> of the requirements for OE is git (or tar) to actually get OE in the first
> place.
>
> I see this as more something some one would run before cloning OE or Poky.
> It’s a replacement for needing to read a long HOWTO on a webpage to get
> started with OE or YP.
>
> Behan
> --
> ___
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-devel
>
-- 
___
Openembedded-devel mailing list
Openembedded-devel@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

Re: [oe] Alpha release of check-requirement.sh

[Behan Webster]

> On Aug 9, 2017, at 7:22 AM, Leonardo Sandoval 
>  wrote:
> 
> in my opinion, this fits nicely into the oe-core, into the
> scripts/contrib folder.

Keeping it in script/contrib seems odd to me (horse before the cart). One of 
the requirements for OE is git (or tar) to actually get OE in the first place.

I see this as more something some one would run before cloning OE or Poky. It’s 
a replacement for needing to read a long HOWTO on a webpage to get started with 
OE or YP.

Behan
-- 
___
Openembedded-devel mailing list
Openembedded-devel@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

Re: [oe] Alpha release of check-requirement.sh

[Behan Webster]

> On Aug 9, 2017, at 11:32 AM, Randy MacLeod  
> wrote:
> 
> On 2017-08-09 10:22 AM, Leonardo Sandoval wrote:
>> in my opinion, this fits nicely into the oe-core, into the
>> scripts/contrib folder.
> 
> Agreed.

As far as living in scripts/contrib, you are certainly free to keep a copy 
there, but as it says on the webpage there is nothing specific to YP nor OE in 
it at the moment (other than the list of requirements), and I will likely be 
adding requirements for other projects.

And what I’m saying is that maintaining it is kinda what I’m already doing as a 
part of my course work (check-requirements is a side effect of that work).

> It would be nice to separate the data from the script so
> that people could easily add to it. That might make updates
> more difficult if the script is kept separate from oe-core.

Well, what you aren’t seeing is that data is separate; it’s all in arrays by 
requirement grouping. The script is also constructed from a series of smaller 
scripts catted together by a makefile; what you’re seeing is the output of a 
script build. I still need to push the git repo to github.

> Could you get rid of the references to 'course' if the script
> gets integrated:
> $ grep -i course ~/Downloads/check-requirements.sh | wc -l
> 139
> 
> s/course/activity/ perhaps.

And the “course” part is current attribute of the shared nature of the 
codebase; mostly variable names. I’d like to make that more generic, but it 
doesn’t hurt much right now. “Activity” is surely a possibility.

Please note I said it was an alpha release. I literally banged this together 
from ready-for.sh in about 8 hours. It is a work in progress. But worth sharing 
considering it’s pretty useful already.

I’d mostly be looking for bug reports as to packages/requirements to add to 
lists for various versions of distros.

It does occur to me that there may need to be a version of this per release of 
YP, but that’s for the future.

Behan
-- 
___
Openembedded-devel mailing list
Openembedded-devel@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-devel

Re: [oe] Alpha release of check-requirement.sh

[Randy MacLeod]

On 2017-08-09 10:22 AM, Leonardo Sandoval wrote:

in my opinion, this fits nicely into the oe-core, into the
scripts/contrib folder.


Agreed.

It would be nice to separate the data from the script so
that people could easily add to it. That might make updates
more difficult if the script is kept separate from oe-core.

Could you get rid of the references to 'course' if the script
gets integrated:
$ grep -i course ~/Downloads/check-requirements.sh | wc -l
139

s/course/activity/ perhaps.

Thanks,

../Randy





On Tue, 2017-08-08 at 17:24 -0700, Behan Webster wrote:

I maintain a script (ready-for.sh) which checks requirements for various 
courses I teach which makes sure you have the right computer, RAM, disk space 
available, and such in order to do the class. It also makes sure that the 
required packages for your $DISTRO are installed (supported distros are CentOS, 
Debian, Fedora, Mint, openSUSE, RHEL, SLES, Ubuntu, with some initial support 
for ArchLinux). It has a full test suite to make sure the package lists are 
valid and available for each of the distro. It also can output a JSON version 
of the requirements (if you want to extract requirements for something else 
like a webpage or document) and includes an auto-updater. Its designed to make 
getting ready for a specific course very easy for a new person. I also use it 
to debug setup issues on people’s computers over the phone or email.

One of the courses I maintain requirements for is the LF Yocto Project course. 
And recently I rebuilt the script into something more generic 
(check-requirements.sh) which specifically allows you to check for requirements 
for the various groupings of packages in the Yocto Project documentation.

I’ve made it initially available as an alpha release via 
https://cm.converseincode.com/cr/  , and I 
plan to maintain it along with the script described above since they share so much of 
the same code.

The idea is that users would download and run this script once in order to make 
sure their machines are ready to build OE or Poky or whatever, instead of 
reading the setup document and manually installing all the packages. One would 
still have to clone the appropriate git repo to get a copy of OE or Poky after 
that.

As an example, here it is being run on an Ubuntu-16.04 machine which is missing 
packages for building the documentation. (Showing the YP requirements are 
boring, because this computer already meets those requirements because I build 
OE all the time). It is to be run as a normal user, but uses sudo internally 
when it needs to run privileged commands. In fact if you try to run it as root, 
it complains.

$ sudo ./check-requirements.sh
[sudo] password for behanw:
FAIL: Please don't run as root
NOTE: Sudo will be used internally by this script as required.

$ ./check-requirements.sh --list
Available (4) options:
   YPDOC - Requirements for Yocto Project Documentation
   YPGUI - Requirements for Yocto Project GUI
   YP - Requirements for Yocto Project
   YPST - Requirements for OpenEmbedded Self-Test

$ ./check-requirements.sh YPDOC
Checking that this computer is suitable for YPDOC: Requirements for Yocto 
Project Documentation

PASS: CPU architecture is x86_64
PASS: 4 core CPU (good enough but 8 is preferred)
PASS: Your CPU appears powerful enough (currently at 23228 BogoMIPS 
cumulatively)
PASS: 6 GiB RAM
PASS: 158 GiB free disk space in /home/behanw
PASS: Linux distribution architecture is x86_64
PASS: Linux distribution is Ubuntu:amd64-16.04 (xenial)
PASS: Internet is available (which is required in this case)

WARN: Some packages are missing
NOTE: Need to install: dblatex docbook-dsssl docbook-utils fop jadetex 
java-wrappers libavalon-framework-java libbatik-java libbsf-java 
libfontbox-java libfop-java libosp5 libostyle1c2 libsgmls-perl libsp1c2 
libxmlgraphics-commons-java openjade sgmlspl sp texlive-bibtex-extra 
texlive-math-extra xmlto
You can install missing packages by running:
   ./check-requirements.sh --install YPDOC
or by:
   sudo apt-get --no-install-recommends install dblatex docbook-dsssl 
docbook-utils fop jadetex java-wrappers libavalon-framework-java libbatik-java 
libbsf-java libfontbox-java libfop-java libosp5 libostyle1c2 libsgmls-perl 
libsp1c2 libxmlgraphics-commons-java openjade sgmlspl sp texlive-bibtex-extra 
texlive-math-extra xmlto

WARN: Your computer doesn't meet the stated requirements unless you can fix the 
above warnings.
WARN: You also have some missing packages.

$ ./check-requirements.sh --install YPDOC
Checking that this computer is suitable for YPDOC: Requirements for Yocto 
Project Documentation


-

Re: [oe] Alpha release of check-requirement.sh

[Leonardo Sandoval]

in my opinion, this fits nicely into the oe-core, into the
scripts/contrib folder.



On Tue, 2017-08-08 at 17:24 -0700, Behan Webster wrote:
> I maintain a script (ready-for.sh) which checks requirements for various 
> courses I teach which makes sure you have the right computer, RAM, disk space 
> available, and such in order to do the class. It also makes sure that the 
> required packages for your $DISTRO are installed (supported distros are 
> CentOS, Debian, Fedora, Mint, openSUSE, RHEL, SLES, Ubuntu, with some initial 
> support for ArchLinux). It has a full test suite to make sure the package 
> lists are valid and available for each of the distro. It also can output a 
> JSON version of the requirements (if you want to extract requirements for 
> something else like a webpage or document) and includes an auto-updater. Its 
> designed to make getting ready for a specific course very easy for a new 
> person. I also use it to debug setup issues on people’s computers over the 
> phone or email.
> 
> One of the courses I maintain requirements for is the LF Yocto Project 
> course. And recently I rebuilt the script into something more generic 
> (check-requirements.sh) which specifically allows you to check for 
> requirements for the various groupings of packages in the Yocto Project 
> documentation.
> 
> I’ve made it initially available as an alpha release via 
> https://cm.converseincode.com/cr/  , and I 
> plan to maintain it along with the script described above since they share so 
> much of the same code.
> 
> The idea is that users would download and run this script once in order to 
> make sure their machines are ready to build OE or Poky or whatever, instead 
> of reading the setup document and manually installing all the packages. One 
> would still have to clone the appropriate git repo to get a copy of OE or 
> Poky after that.
> 
> As an example, here it is being run on an Ubuntu-16.04 machine which is 
> missing packages for building the documentation. (Showing the YP requirements 
> are boring, because this computer already meets those requirements because I 
> build OE all the time). It is to be run as a normal user, but uses sudo 
> internally when it needs to run privileged commands. In fact if you try to 
> run it as root, it complains.
> 
> $ sudo ./check-requirements.sh 
> [sudo] password for behanw: 
> FAIL: Please don't run as root
> NOTE: Sudo will be used internally by this script as required.
> 
> $ ./check-requirements.sh --list
> Available (4) options:
>   YPDOC - Requirements for Yocto Project Documentation
>   YPGUI - Requirements for Yocto Project GUI
>   YP - Requirements for Yocto Project
>   YPST - Requirements for OpenEmbedded Self-Test
> 
> $ ./check-requirements.sh YPDOC
> Checking that this computer is suitable for YPDOC: Requirements for Yocto 
> Project Documentation
> 
> PASS: CPU architecture is x86_64
> PASS: 4 core CPU (good enough but 8 is preferred)
> PASS: Your CPU appears powerful enough (currently at 23228 BogoMIPS 
> cumulatively)
> PASS: 6 GiB RAM
> PASS: 158 GiB free disk space in /home/behanw
> PASS: Linux distribution architecture is x86_64
> PASS: Linux distribution is Ubuntu:amd64-16.04 (xenial)
> PASS: Internet is available (which is required in this case)
> 
> WARN: Some packages are missing
> NOTE: Need to install: dblatex docbook-dsssl docbook-utils fop jadetex 
> java-wrappers libavalon-framework-java libbatik-java libbsf-java 
> libfontbox-java libfop-java libosp5 libostyle1c2 libsgmls-perl libsp1c2 
> libxmlgraphics-commons-java openjade sgmlspl sp texlive-bibtex-extra 
> texlive-math-extra xmlto
> You can install missing packages by running:
>   ./check-requirements.sh --install YPDOC
> or by:
>   sudo apt-get --no-install-recommends install dblatex docbook-dsssl 
> docbook-utils fop jadetex java-wrappers libavalon-framework-java 
> libbatik-java libbsf-java libfontbox-java libfop-java libosp5 libostyle1c2 
> libsgmls-perl libsp1c2 libxmlgraphics-commons-java openjade sgmlspl sp 
> texlive-bibtex-extra texlive-math-extra xmlto
> 
> WARN: Your computer doesn't meet the stated requirements unless you can fix 
> the above warnings.
> WARN: You also have some missing packages.
> 
> $ ./check-requirements.sh --install YPDOC
> Checking that this computer is suitable for YPDOC: Requirements for Yocto 
> Project Documentation
> 
> 
> 
> WARN: Some packages are missing
> WARN: About to install: dblatex docbook-dsssl docbook-utils fop jadetex 
> java-wrappers libavalon-framework-java libbatik-java libbsf-java 
> libfontb