Re: [yocto] Downloading Git Archives

[Chuck Wolber]

Thanks for replying!

That is effectively what we have to do with the VM clone of the build
machine. We spin up the VM, do the build, see what is new in the downloads
directory, do our due diligence, and push the new stuff into the local
mirror behind the firewall. The primary build server must remain "sterile",
so everything has to be "hoisted" over the fence manually.

[Side Note: We also maintain a local clone of the openembedded and poky
repositories. Our build pins to a specific hash, which gets advanced after
we do our due diligence. This is when we discover that certain packages
need to be added to our ${LOCAL_MIRROR}. My desire is to automate as much
as possible, while exposing as little as possible.]

Whenever the nightly build fails due to a missing package, and I see a URL
with a (for example) .tar.gz in the build error log, I know my job is a lot
easier. I can simply wget from any Internet connected host, instead of
spinning up the VM clone, waiting for a build to complete, and then
figuring out what is new in the downloads directory.

I was hoping that there was an equally simple way of grabbing git
repository packages from an Internet connected host without having a
bitbake build cache them for me.

I understand that these packages are effectively a bare clone, but there
seems to be some additional non-obvious (to me at least) "magic" that makes
it more complicated than doing a git repo bare clone and wrapping it up
into a .tar.gz.

I am happy to start trawling through source code and figure it out on my
own, but I figured this had to be common knowledge to someone on this list.

..Ch:W..

On Thu, Sep 26, 2019 at 8:50 PM Rudolf J Streif 
wrote:

> Chuck,
>
> Unless I am missing something (which I cannot entirely rule out), the
> default PREMIRRORS should actually do what you want. For the poky distro
> configuration has this:
>
> PREMIRRORS ??= "\
> bzr://.*/.*   http://downloads.yoctoproject.org/mirror/sources/ \n \
> cvs://.*/.*   http://downloads.yoctoproject.org/mirror/sources/ \n \
> git://.*/.*   http://downloads.yoctoproject.org/mirror/sources/ \n \
> gitsm://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n \
> hg://.*/.*http://downloads.yoctoproject.org/mirror/sources/ \n \
> osc://.*/.*   http://downloads.yoctoproject.org/mirror/sources/ \n \
> p4://.*/.*http://downloads.yoctoproject.org/mirror/sources/ \n \
> svn://.*/.*   http://downloads.yoctoproject.org/mirror/sources/ \n"
>
> This causes the fetcher to intercept and translate all the SCM protocols
> into http downloads from the YP mirror. Since you are saying wget works for
> you this should actually automatically work for you. If you build system
> can at least download using http then
>
> bitbake  --runall=fetch
>
> should download all the sources for you. Then you can disconnect the box
> from the Internet and start the build offline.
>
> :rjs
> On 9/26/19 8:06 PM, Chuck Wolber wrote:
>
> Hi All,
>
> Our build environment is stuck behind a firewall with no access to the
> Internet. When a Yocto build requires a new package, we normally extract
> the URL from the failed build log and "wget" the package from the upstream
> source, do our due diligence (checksums, etc), and then add it to the
> server pointed to by ${LOCAL_MIRROR}.
>
> Obviously that procedure does not work so well when a git repository needs
> to be fetched. We have tried various methods, but nothing seems to work
> reliably. Our current method is an annoying hack involving a virtual
> machine clone of the build server.
>
> I have searched far and wide, and cannot find any sort of procedure. Can
> someone point me at a procedure for translating something like git://
> git.yoctoproject.org/pseudo into a file we can store in our
> ${LOCAL_MIRROR}?
>
> Thank you,
>
> ..Ch:W..
>
> --
> -
> Rudolf J Streif
> CEO/CTO ibeeto
> +1.855.442.3386 x700
>
>

-- 
“I would challenge anyone here to think of a question upon which we once
had a scientific answer, however inadequate, but for which now the best
answer is a religious one."  -Sam Harris
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

[yocto] Downloading Git Archives

[Chuck Wolber]

Hi All,

Our build environment is stuck behind a firewall with no access to the
Internet. When a Yocto build requires a new package, we normally extract
the URL from the failed build log and "wget" the package from the upstream
source, do our due diligence (checksums, etc), and then add it to the
server pointed to by ${LOCAL_MIRROR}.

Obviously that procedure does not work so well when a git repository needs
to be fetched. We have tried various methods, but nothing seems to work
reliably. Our current method is an annoying hack involving a virtual
machine clone of the build server.

I have searched far and wide, and cannot find any sort of procedure. Can
someone point me at a procedure for translating something like git://
git.yoctoproject.org/pseudo into a file we can store in our ${LOCAL_MIRROR}?

Thank you,

..Ch:W..
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] Yocto System Requirement for developing OS

[Chuck Wolber]

It depends on a lot of factors. But for actually building yocto images... I
find that 1.5 gigs of RAM per CPU seems to be about right on the virtual
machines we use to build images.

..Ch:W..

On Fri, Apr 26, 2019 at 17:29 raja.g kumar  wrote:

> Hi
> Can I know the RAM ,Disk space and processor requirement for developing
> the os with yocto for IoT ?
>
> Please suggest and it would be greatly helpful.
>
> Regards
> Rajakumar G
>
> Get Outlook for Android 
>
> --
> ___
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>
-- 
“I would challenge anyone here to think of a question upon which we once
had a scientific answer, however inadequate, but for which now the best
answer is a religious one."  -Sam Harris
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] Storing Sstate in S3 success stories?

[Chuck Wolber]

Have you done any wireshark analysis on the traffic? My guess is that the
round trip with network latency is bumping your build time by a factor of
at least 100x.  The state-cache is hammered on continuously, so have
probably introduced a significant bottleneck.

..Ch:W..

On Mon, Feb 25, 2019 at 17:45 Timothy Froehlich 
wrote:

> I've been spending a bit too long this past week trying to build up a
> reproducable build infrastructure in AWS and I've got very little
> experience with cloud infrastucture and I'm wondering if I'm going in the
> wrong direction. I'm attempting to host my sstate_cache as a mirror in a
> private S3 bucket, and I believe I have everything configured properly,
> including exposing the bucket to http requests, since I can wget files that
> I've previously synced up to the bucket. However if I add in the
> SSTATE_MIRRORS to my build, bitbake slows to a crawl (it's a powerful VM)
> and barely seems to get anything. The EC2 instance is in the same region as
> the S3 bucket, roles have been configured properly to allow access, etc.
>
> I'm not looking for help debugging this, I just want to know whether I'm
> right that hosting my sstate in an S3 bucket should work. I've only been
> able to find one mention of it being done with no reproduction hints.
>
> --
> ___
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>
-- 
“I would challenge anyone here to think of a question upon which we once
had a scientific answer, however inadequate, but for which now the best
answer is a religious one."  -Sam Harris
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] npm recipe bad task order

[Chuck Wolber]

Normally the source archive is unpacked first, which contains the license
file. Is there something different with the source packaging for your
recipe?

..Ch:W..

On Thu, Feb 14, 2019 at 02:26 Jonas Andersson 
wrote:

> Hi,
>
> I have some recipes created from npm packages with devtool.
>
> do_populate_lic with: WARNING: Could not copy license file
> and LIC_FILES_CHKSUM points to an invalid file.
>
> In my understanding the problem is that do_populate_lic task is running
> before do_compile:
>
> log.task_order
> ___
> do_cleansstate (20550): log.do_cleansstate.20550
> do_cleanall (20551): log.do_cleanall.20551
> do_fetch (20653): log.do_fetch.20653
> do_unpack (22861): log.do_unpack.22861
> do_prepare_recipe_sysroot (22862): log.do_prepare_recipe_sysroot.22862
> do_patch (23281): log.do_patch.23281
> do_configure (23283): log.do_configure.23283
> do_populate_lic (23284): log.do_populate_lic.23284
> do_compile (23289): log.do_compile.23289
> ___
>
> The recipes npm created with devtool reference license to S
> (${WORKDIR}/npmpkg):
>
> __
> LIC_FILES_CHKSUM = "file://LICENSE;md5=75dbbb2f49d90bfc476532dc5861fd26 \
>
> file://node_modules/inherits/LICENSE;md5=5b2ef2247af6d355ae9d9f988092d470 \
>
> file://node_modules/psl/LICENSE;md5=2425d288296fae32e27553694ff40294 \
>
> file://node_modules/rc/LICENSE.MIT;md5=e0f70a42adf526e6f5e605a94d98a420"
>
> __
>
> But as npm_do_compile is the task that populates workdir there is no
> license files available when do_populate_lic runs.
>
> I have tried both sumo and thud but with the same result, is the order
> something that have change?
>
> Best regards
> Jonas
>
> --
> ___
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>
-- 
“I would challenge anyone here to think of a question upon which we once
had a scientific answer, however inadequate, but for which now the best
answer is a religious one."  -Sam Harris
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] icecast

[Chuck Wolber]

You have to refactor xslt-config out of the autotools macros and use
pkg-config instead.

This recipe is an example:

meta/recipes-devtools/swig/swig/0001-configure-use-pkg-config
-for-pcre-detection.patch

..Ch:W..

On Thu, Feb 14, 2019 at 05:57 Leonardo Jose Duarte MendesJunior <
leodmende...@gmail.com> wrote:

> I'm trying to compile the package icecast. The package icecast is old. The
> package depends on xslt.
> When I compile this package, I had the issue with xslt.
> How to use pkgconfig if the own source code use him ?
> --
> ___
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>
-- 
“I would challenge anyone here to think of a question upon which we once
had a scientific answer, however inadequate, but for which now the best
answer is a religious one."  -Sam Harris
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] Two recipes install the same file

[Chuck Wolber]

I have run across this a few times, particularly with man pages, when
including upstream packages into my images.

I have to use a bbappend recipe in my meta layer to remove the lower
priority version.

I quite like the feature because it ensures that all collisions are known
and dealt with in a clear and formal way.

..Ch:W..

On Thu, Feb 14, 2019 at 08:55 Burton, Ross  wrote:

> Two packages can't install the same file.  In this case I suspect
> you'll need to split up the packaging further.  A common alternative
> for binaries in /bin which want to be installed by both e.g. coreutils
> and busybox is to use update-alternatives.
>
> Ross
>
> On Thu, 14 Feb 2019 at 16:36, madoga  wrote:
> >
> > Hello everyone,
> >
> > I have an issue while I'm trying to overwrite some files that two
> recipes are installing. Task do_rootfs fails and I want to ask about the
> best way to solve this:
> >
> > -
> > Error: Transaction check error:
> >   file /app/econfig.xml conflicts between attempted installs of
> tests-1.0-r0 and error-1.0-r0
> >   file /app/controller.xml conflicts between attempted installs of
> tests-1.0-r0 and controller-1.0-r0
> >   file /app/user_mode.xml conflicts between attempted installs of
> mode-1.0-r0 and tests-1.0-r0
> > Error Summary
> > -
> >  When I launch bitbake with a concrete image this error appears and I
> don't know how I can give "priority" to a recipe. I have found something
> about create .bbappend file...
> >
> > Thank you,
> > Mario
> >
> >
> >
> >
> > --
> > ___
> > yocto mailing list
> > yocto@yoctoproject.org
> > https://lists.yoctoproject.org/listinfo/yocto
> --
> ___
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>
-- 
“I would challenge anyone here to think of a question upon which we once
had a scientific answer, however inadequate, but for which now the best
answer is a religious one."  -Sam Harris
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] XSLT-CONFIG ERROR

[Chuck Wolber]

You have to refactor the Icecast bitbake recipe to use pkg-config instead
of xslt-config.

This patch might give you some ideas:
meta/recipes-devtools/swig/swig/0001-configure-use-pkg
-config-for-pcre-detection.patch

I patched modsecurity to fix the same issue. It mostly works, but their
build fails in certain configurations. Feel free to check my patch if you
dare, but I am not an autotools expert, and probably did something wrong.

https://github.com/SpiderLabs/ModSecurity/pull/2014/files

..Ch:W..

On Mon, Feb 4, 2019 at 10:35 AM Igor Miranda 
wrote:

> Hey guys, I'm trying to compile the Icecast, but I'm getting the
> xslt-config error, which says that I need to change to pkg-config, but I
> dont know how to do this. Should I do a patch? Or its more easy to change
> something in the Yocto to enable the use of the xslt-config? If I need to
> use a patch, can someone explain to me how do I do something or can someone
> send me a patch that works. Please I dont have more time, I need to
> give this project, working, as soon as possible. HELP ME!
> --
> ___
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>


-- 
“I would challenge anyone here to think of a question upon which we once
had a scientific answer, however inadequate, but for which now the best
answer is a religious one."  -Sam Harris
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] pcre-config replacement

[Chuck Wolber]

On Thu, Jan 31, 2019 at 2:14 AM Burton, Ross  wrote:

> pkg-config.  Inherit the pkgconfig class in the recipe if it doesn't
> already, and see for example
>
> meta/recipes-devtools/swig/swig/0001-configure-use-pkg-config-for-pcre-detection.patch
> in oe-core for an example.
>
> This is an upstreamable change so ideally submit it upstream so you
> don't have to carry a patch around too.
>

Indeed, that is exactly what I have done:
https://github.com/SpiderLabs/ModSecurity/pull/2014

It looks like a few of their automated regression tests are failing, so I
probably have more work to do on the patch.

I did end up finding the patch you referenced, and that got me most of the
way there. I do not use autotools much, so it is hard for me to tell if I
missed something obvious.

..Ch:W..
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

[yocto] pcre-config replacement

[Chuck Wolber]

What is the recommended packaging approach for an upstream configure script
that seems to think that it must have pcre-config (which is disabled by the
binconfig-disabled class)?

Looking at the source of the upstream package seems to indicate that they
actually do need pcre, so this is not as simple as patching the configure
script.

..Ch:W..
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

[yocto] Deactivating KERNEL_FEATURES

[Chuck Wolber]

This question is with respect to the Yocto thud (2.6.x) branch. I have also
reviewed section 2.3.3 and 2.3.4 of the Kernel Development document
.


My kernel configuration (cfg) seems to be overridden by KERNEL_FEATURES and
it is not clear how to stop this behavior. To take one specific example,
nothing I have tried seems to successfully deactivate the sound
configuration kernel options.

My linux-yocto_4.18.%.bbappend file has the following contents:

FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI_append += "file://0001-kernel-config.cfg"

The following line appears in 0001-kernel-config.cfg:

# CONFIG_SOUND is not set

And yet when I configure the kernel (bitbake -c configure linux-yocto) the
resultant .config in the linux-yocto ${WORKDIR} contains "CONFIG_SOUND=m"
and a plethora of other sound related options that should be turned off. I
did the full kernel build and verified that the .config reflects the final
result in the kernel.

"bitbake -e linux-yocto" contains the following, which seems to explain a
bit:

KERNEL_FEATURES=" features/nfsd/nfsd-enable.scc features/debug/printk.scc
 features/kernel-sample/kernel-sample.scc features/netfilter/netfilter.scc
cfg/virtio.scc cfg/sound.scc cfg/paravirt_kvm.scc "

I should also add that our configuration includes:

DISTRO_FEATURES_remove = " alsa pulseaudio"
DISTRO_FEATURES_BACKFILL_CONSIDERED = " alsa pulseaudio".


How do I ensure that my kernel config gets the last say and is not
overridden?

Thank you,

..Ch:W..
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] Yocto Thud - /lib64 link - Build Failure

[Chuck Wolber]

Thank you, switching to IMAGE_PREPROCESS_COMMAND_append seems to have
solved the problem.

..Ch:W..

On Wed, Nov 28, 2018 at 7:14 PM ChenQi  wrote:

> This is because prelink_image is also in IMAGE_PREPROCESS_COMMAND.
>
> meta/classes/image-prelink.bbclass:IMAGE_PREPROCESS_COMMAND_append_libc-glibc
> = " prelink_setup; prelink_image; "
>
> And as it's using _append, it will be after your 'symlink_lib64'.
> Maybe using IMAGE_PREPROCESS_COMMAND_append = " symlink_lib64; " would
> work for you?
>
> Best Regards,
> Chen Qi
>
> On 11/29/2018 08:09 AM, Chuck Wolber wrote:
>
> Background: I have an image that works in pyro. I am migrating the image
> to thud.
>
> Some of the add-on code to our image requires the /lib64 directory (in
> particular the VirtualBox Additions). I have proven that a simple symlink
> (/lib -> /lib64) fixes the problem. This symlink was being applied with the
> following code at the very bottom of the image:
>
> IMAGE_PREPROCESS_COMMAND += "symlink_lib64; "
>
> symlink_lib64() {
>
> ln -s /lib ${IMAGE_ROOTFS}/lib64
>
> }
>
>
> When I use that same code with Yocto thud, I get the following build error:
>
> DEBUG: Executing shell function prelink_image
>
> Size before prelinking 818112.
>
> /build/poky/tmp/work/qemux86_64-linux/common-os-image/1.0-r0/recipe-sysroot-native/usr/sbin/prelink:
> Could not record directory /lib/modules-load.d: No such file or directory
>
> WARNING: exit code 1 from a shell command.
>
>
> Looking at the image generation diagram here (
> https://www.yoctoproject.org/docs/2.6/overview-manual/overview-manual.html#image-generation-dev-environment),
> it would seem to indicate that IMAGE_PREPROCESS_COMMAND runs after the
> linker steps in do_rootfs. So it is unclear why prelink is failing during
> the do_image phase.
>
> Where is the correct place to create a /lib -> /lib64 symlink in our image
> recipe?
>
> Thank you,
>
> ..Ch:W..
>
>
>
>

-- 
“I would challenge anyone here to think of a question upon which we once
had a scientific answer, however inadequate, but for which now the best
answer is a religious one."  -Sam Harris
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

[yocto] Yocto Thud - /lib64 link - Build Failure

[Chuck Wolber]

Background: I have an image that works in pyro. I am migrating the image to
thud.

Some of the add-on code to our image requires the /lib64 directory (in
particular the VirtualBox Additions). I have proven that a simple symlink
(/lib -> /lib64) fixes the problem. This symlink was being applied with the
following code at the very bottom of the image:

IMAGE_PREPROCESS_COMMAND += "symlink_lib64; "

symlink_lib64() {

ln -s /lib ${IMAGE_ROOTFS}/lib64

}


When I use that same code with Yocto thud, I get the following build error:

DEBUG: Executing shell function prelink_image

Size before prelinking 818112.

/build/poky/tmp/work/qemux86_64-linux/common-os-image/1.0-r0/recipe-sysroot-native/usr/sbin/prelink:
Could not record directory /lib/modules-load.d: No such file or directory

WARNING: exit code 1 from a shell command.


Looking at the image generation diagram here (
https://www.yoctoproject.org/docs/2.6/overview-manual/overview-manual.html#image-generation-dev-environment),
it would seem to indicate that IMAGE_PREPROCESS_COMMAND runs after the
linker steps in do_rootfs. So it is unclear why prelink is failing during
the do_image phase.

Where is the correct place to create a /lib -> /lib64 symlink in our image
recipe?

Thank you,

..Ch:W..
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto