Re: [Qbs] Using ConanfileProbe with yocto cross compiling toolchain

[Raphael Cotty]

Hi,
I pushed some scripts in issue QBS-1188
.
They allow to compile qbs inside yocto and also to use qbs to compile
inside a recipe (just by inheriting from qbs).

But when developing code using qtc this is not very convenient so I have a
script that generates this kind of profile:
qt-project\qbs\profiles\qtc_oe_25deb62a\cpp\cCompilerName=gcc
qt-project\qbs\profiles\qtc_oe_25deb62a\cpp\compilerName=g++
qt-project\qbs\profiles\qtc_oe_25deb62a\cpp\cxxCompilerName=g++
qt-project\qbs\profiles\qtc_oe_25deb62a\cpp\platformCommonCompilerFlags=@Invalid()
qt-project\qbs\profiles\qtc_oe_25deb62a\cpp\platformLinkerFlags=@Invalid()
qt-project\qbs\profiles\qtc_oe_25deb62a\cpp\toolchainInstallPath=/usr/local/oecore-x86_64/sysroots/x86_64-cascadesdk-linux/usr/bin/arm-cascade-linux-gnueabi
qt-project\qbs\profiles\qtc_oe_25deb62a\cpp\toolchainPrefix=arm-cascade-linux-gnueabi-
qt-project\qbs\profiles\qtc_oe_25deb62a\moduleProviders\Qt\qmakeFilePaths=/home/raphael/src/yocto/rocko/poky/meta-cascade/script/qmake
qt-project\qbs\profiles\qtc_oe_25deb62a\qbs\architecture=arm
qt-project\qbs\profiles\qtc_oe_25deb62a\qbs\sysroot=/usr/local/oecore-x86_64/sysroots/cortexa9hf-neon-cascade-linux-gnueabi
qt-project\qbs\profiles\qtc_oe_25deb62a\qbs\targetPlatform=linux
qt-project\qbs\profiles\qtc_oe_25deb62a\qbs\toolchain=@Variant(\0\0\0\t\0\0\0\x1\0\0\0\n\0\0\0\x6\0g\0\x63\0\x63)

I've attached the different scripts.
It's not very clean but allows to use qbs/qtc without having to source the
yocto env.

Having a qbs-setup-yocto script/program could be used to achieve that.
This tool should be provided by qbs like all other qbs-setup-*.
I don't know what should be done about the qbs recipes. Trying to add them
in meta-qt?

Raph

Le sam. 23 mai 2020 à 03:34, Christian Gagneraud  a
écrit :

> On Sat, 23 May 2020 at 07:34, Richard Weickelt 
> wrote:
> > > Thus I don't think that it should be made into the official Qbs
> codebase.
> > > Maybe this code could be used in some How-To section or something.
> >
> > What's your experience in using Qbs with Yocto? Once you solved the
> > EnvironmentScriptProbe, were there a lot of additional steps needed? Any
> > pitfalls? I guess a howto would be nice in general and might be
> interesting
> > for some people. On the other hand - and without having much knowledge
> about
> > yocto - how high is the risk that this howto would be outdated by
> tomorrow?
>
> Yocto is a "classic" for embedded Linux system. It's been around for
> many years (ages).
> It is a REFERENCE in this domain, is sponsored by the Linux
> foundation, and SoC manufacturers contribute mainstream by providing
> their own "hardware layer".
> It support all common (and uncommon) architectures used in the
> industry (PowerPC, Mips, Arm, ...)
> Last but not least, Yocto is the technology behind Boot2Qt, Qt5 has
> it's own layer:
> https://www.yoctoproject.org/software-overview/layers/?searchTerm=meta-qt5
> ,
> and boot2qt-opensource too:
> https://code.qt.io/cgit/yocto/meta-boot2qt.git/
>
> Adding support for Yocto to Qbs is IMHO a big win.
> Unfortunately using Yocto (aka. Poky, OpenEmbedded, ...) starts with
> sourcing an env file.
> An alternative to this probe would be a to make the qbs-setup-* tools
> do the right thing when run from a shell which has sourced that file.
> That will allow to setup different profiles for different version of
> Yocto (typ. stable vs dev) and name them accordingly.
>
> Call me crazy, but i would say that a qbs-setup-yocto tool that takes
> a path to an env file would be a good idea, or something similar to
> qbs-setup-android.
>
> FYI, this is the sort of things the env file defines:
>
> https://www.yoctoproject.org/docs/2.3.3/sdk-manual/sdk-manual.html#sdk-running-the-extensible-sdk-environment-setup-script
>
> IMHO, it is worth having a quick look at the "app developer workflow":
>
> https://www.yoctoproject.org/docs/2.3.3/sdk-manual/sdk-manual.html#sdk-use-devtool-to-add-an-application
> Haven't tried, but i'm suspecting QtC integrates this somehow.
>
> Chris
> ___
> Qbs mailing list
> Qbs@qt-project.org
> https://lists.qt-project.org/listinfo/qbs
>


qt.conf
Description: Binary data


qmake
Description: Binary data


post_yocto_install.sh
Description: application/shellscript


qt5.conf
Description: Binary data
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Using ConanfileProbe with yocto cross compiling toolchain

[Christian Gagneraud]

On Sat, 23 May 2020 at 07:34, Richard Weickelt  wrote:
> > Thus I don't think that it should be made into the official Qbs codebase.
> > Maybe this code could be used in some How-To section or something.
>
> What's your experience in using Qbs with Yocto? Once you solved the
> EnvironmentScriptProbe, were there a lot of additional steps needed? Any
> pitfalls? I guess a howto would be nice in general and might be interesting
> for some people. On the other hand - and without having much knowledge about
> yocto - how high is the risk that this howto would be outdated by tomorrow?

Yocto is a "classic" for embedded Linux system. It's been around for
many years (ages).
It is a REFERENCE in this domain, is sponsored by the Linux
foundation, and SoC manufacturers contribute mainstream by providing
their own "hardware layer".
It support all common (and uncommon) architectures used in the
industry (PowerPC, Mips, Arm, ...)
Last but not least, Yocto is the technology behind Boot2Qt, Qt5 has
it's own layer:
https://www.yoctoproject.org/software-overview/layers/?searchTerm=meta-qt5,
and boot2qt-opensource too:
https://code.qt.io/cgit/yocto/meta-boot2qt.git/

Adding support for Yocto to Qbs is IMHO a big win.
Unfortunately using Yocto (aka. Poky, OpenEmbedded, ...) starts with
sourcing an env file.
An alternative to this probe would be a to make the qbs-setup-* tools
do the right thing when run from a shell which has sourced that file.
That will allow to setup different profiles for different version of
Yocto (typ. stable vs dev) and name them accordingly.

Call me crazy, but i would say that a qbs-setup-yocto tool that takes
a path to an env file would be a good idea, or something similar to
qbs-setup-android.

FYI, this is the sort of things the env file defines:
https://www.yoctoproject.org/docs/2.3.3/sdk-manual/sdk-manual.html#sdk-running-the-extensible-sdk-environment-setup-script

IMHO, it is worth having a quick look at the "app developer workflow":
https://www.yoctoproject.org/docs/2.3.3/sdk-manual/sdk-manual.html#sdk-use-devtool-to-add-an-application
Haven't tried, but i'm suspecting QtC integrates this somehow.

Chris
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Using ConanfileProbe with yocto cross compiling toolchain

[Richard Weickelt]

> I archived my goal with the following code:

Good to hear. Thanks for posting the code.

> But this code has some restrictions (like using bash or zsh but not fish).

I wouldn't consider it a restriction not to support an exotic shell.

> Thus I don't think that it should be made into the official Qbs codebase.
> Maybe this code could be used in some How-To section or something.

What's your experience in using Qbs with Yocto? Once you solved the
EnvironmentScriptProbe, were there a lot of additional steps needed? Any
pitfalls? I guess a howto would be nice in general and might be interesting
for some people. On the other hand - and without having much knowledge about
yocto - how high is the risk that this howto would be outdated by tomorrow?

> If you really want me to contribute this as a "YoctoEnvironmentProbe",
> despite the restrictions, I will do some cleanup an contribute it.

Probing exported environment variables from a shell script deserves a
generic probe. If you could make it work for .bat and ps1 as well, even
better. Don't forget to add a testcase in test_blackbox.cpp.

Thank you for you contribution.

___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Using ConanfileProbe with yocto cross compiling toolchain

[Kai Dohmen via Qbs]

Hello.

I archived my goal with the following code:

Yocto_Conan_Qbs.qbs:
import qbs
import qbs.File
import qbs.FileInfo
import qbs.Probes
import qbs.Process
import qbs.TextFile
import qbs.Utilities

Project {
references: conanFileProbe.generatedFilesPath + "/conanbuildinfo.qbs"

Probe {
id: yoctoEnvProbe

/* in */
property path environmentSetupScriptPath:
"/home/psy-kai/Develop/Yocto_SDKs/armv7hf_reference/environment-setup-armv7vet2hf-neon-poky-linux-gnueabi"

/* out */
property var environmentVariables

/* private */
readonly property path _projectBuildDirectory:
project.buildDirectory
readonly property path _envExportScript:
FileInfo.cleanPath(FileInfo.joinPaths(project.sourceDirectory,
"env_export.sh"))

configure: {
var extractEnvironmentVariables = function(textFile){
var extractVariable = function(line){
var eqPos = line.indexOf("=");
var name = line.substr(0, eqPos);
var value = line.substr(eqPos+1);
if (!name)
throw "Invalid line in "+textFile.filePath();
return {"name": name, "value": value};
}

var envVars = [];
while (!textFile.atEof())
envVars.push(extractVariable(textFile.readLine()));
textFile.close();
return envVars;
}

var outputEnvFiles = FileInfo.cleanPath(
FileInfo.joinPaths(_projectBuildDirectory,
"yoctoenv",

 Utilities.getHash(environmentSetupScriptPath)));
File.makePath(outputEnvFiles);
var emptyEnvFilePath = FileInfo.joinPaths(outputEnvFiles,
"emptyEnv");
var yoctoEnvFilePath = FileInfo.joinPaths(outputEnvFiles,
"yoctoEnv");
var process = new Process();
try {
process.exec("env",
 ["-", _envExportScript,
  environmentSetupScriptPath, emptyEnvFilePath,
yoctoEnvFilePath],
 true);
} finally {
process.close();
}

var emptyEnvVars = extractEnvironmentVariables(new
TextFile(emptyEnvFilePath));
var yoctoEnvVars = extractEnvironmentVariables(new
TextFile(yoctoEnvFilePath));

environmentVariables =
yoctoEnvVars.filter(function(yoctoEnvVar){
for (var i in emptyEnvVars) {
if (emptyEnvVars[i].name === yoctoEnvVar.name)
return false;
}
return true;
});
found = true;
}
}

Probes.ConanfileProbe {
id: conanFileProbe
conanfilePath: project.sourceDirectory+"/conanfile.txt"
additionalArguments: {
var args = ["-pr:b=imx6", "-b=missing"];
for (var i in yoctoEnvProbe.environmentVariables) {
var envVar = yoctoEnvProbe.environmentVariables[i];
args.push("-e:b="+envVar.name+"="+envVar.value);
}
return args
}
generators: "qbs"
}
}

env_export.sh:
#!/bin/sh

env > $2
source $1
env > $3

But this code has some restrictions (like using bash or zsh but not fish).
Thus I don't think that it should be made into the official Qbs codebase.
Maybe this code could be used in some How-To section or something.

If you really want me to contribute this as a "YoctoEnvironmentProbe",
despite the restrictions, I will do some cleanup an contribute it.


Kai

Am Fr., 15. Mai 2020 um 13:50 Uhr schrieb Richard Weickelt <
rich...@weickelt.de>:

>
> > Sorry. I meant how to extract the environment variables from the yocto
> script.
> > My idea was to source the script in a process
> > with https://doc.qt.io/qbs/jsextension-process.html
> > and then just query all the process environment variables with
> > https://doc.qt.io/qbs/jsextension-environment.html#currentenv.
> > This way I would not need to parse the yocto script myself in the
> > "YoctoEnvProbe".
>
> Maybe you can write another script that
> 1. prints the environment variables
> 2. then sources the yocto script
> 3. then prints the environment variables again
>
> Run this script in a Probe (with an appropriate shell) and parse the output
> somehow. You may also omit step 1 and just use all environment variables
> printed by script.
>
> Richard
> ___
> Qbs mailing list
> Qbs@qt-project.org
> https://lists.qt-project.org/listinfo/qbs
>
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Using ConanfileProbe with yocto cross compiling toolchain

[Richard Weickelt]

> Sorry. I meant how to extract the environment variables from the yocto script.
> My idea was to source the script in a process
> with https://doc.qt.io/qbs/jsextension-process.html 
> and then just query all the process environment variables with
> https://doc.qt.io/qbs/jsextension-environment.html#currentenv.
> This way I would not need to parse the yocto script myself in the
> "YoctoEnvProbe".

Maybe you can write another script that
1. prints the environment variables
2. then sources the yocto script
3. then prints the environment variables again

Run this script in a Probe (with an appropriate shell) and parse the output
somehow. You may also omit step 1 and just use all environment variables
printed by script.

Richard
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Using ConanfileProbe with yocto cross compiling toolchain

[Kai Dohmen via Qbs]

> > Do you know how I can easily extract the environment variables?
> >  https://doc.qt.io/qbs/jsextension-environment.html#currentenv  seems
like a
> > solution but I don't know how to access a process environment with this.
>
> Which environment variables? If I understood you correctly, then you have
> environment variables generated by yocto in a file/script which you need
to
> parse somehow in a probe. There is no probe for that, you would have to
> implement your own.

Sorry. I meant how to extract the environment variables from the yocto
script.
My idea was to source the script in a process with
https://doc.qt.io/qbs/jsextension-process.html
and then just query all the process environment variables with
https://doc.qt.io/qbs/jsextension-environment.html#currentenv.
This way I would not need to parse the yocto script myself in the
"YoctoEnvProbe".
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Using ConanfileProbe with yocto cross compiling toolchain

[Richard Weickelt]

> I will try this approach. But maybe I use `conan install -e` instead of
> setting the environment variables for the complete process.

If the result is the same, then why not.

> Do you know how I can easily extract the environment variables?
>  https://doc.qt.io/qbs/jsextension-environment.html#currentenv  seems like a
> solution but I don't know how to access a process environment with this.

Which environment variables? If I understood you correctly, then you have
environment variables generated by yocto in a file/script which you need to
parse somehow in a probe. There is no probe for that, you would have to
implement your own.

If the environment variables are already in the environment when calling qbs
resolve (for instance by sourcing the script in the shell), then
ConanfileProbe should already have them.
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Using ConanfileProbe with yocto cross compiling toolchain

[Kai Dohmen via Qbs]

> 1. Add an additionalEnvironmentVariables property to ConanfileProbe
> 2. Append these to the process environment in ConanfileProbe. See
>https://doc.qt.io/qbs/jsextension-process.html#setenv
> 3. Add an additional Probe item in your project file that extracts
>the environment variables from the script.
>
>   Probes are currently executed in the order of their appearance in the
qbs
>   file, so make sure it comes first.
>
>  Would that be a solution for you? Then I would warmthly welcome a patch.

I will try this approach. But maybe I use `conan install -e` instead of
setting the environment variables for the complete process.
Do you know how I can easily extract the environment variables?
 https://doc.qt.io/qbs/jsextension-environment.html#currentenv  seems like
a solution but I don't know how to access a process environment with this.

Thanks for the help so far.

Kai
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Using ConanfileProbe with yocto cross compiling toolchain

[Christian Gagneraud]

On Fri, 15 May 2020 at 19:46, Kai Dohmen via Qbs  wrote:
>
> Hello,
>
> I just stumbled across a scenario in which we use with multiple profiles for 
> different cross compiling toolchains. Yocto creates a script which exports 
> environment variables. We normally just extract the needed values from the 
> script and set them as additional qbs profile settings.
> This leads to the problem that the ConanfileProbe just calls "conan 
> install... -p..." without the needed environment variables. Thus when install 
> libraries they are getting compiled with the wrong toolchain/settings.
> What is the proper way of doing this? How do I set the environment variables? 
> Should I set the variables in the Kit?

Personal opinion: I hate yocto simply because it requires to run any
of your tool (IDE, build system) under a controlled environment.
Things might have changed a bit, eg Boot2Qt is based on yocto, and you
can use QtC with boo2qt w/o sourcing an env first.
Unfortunately i do not know the details on how they achieved that.

IMHO, using Qbs in a way that makes it heavily shell environment
dependent is going against Qbs spirit. The profile system is one of
Qbs cornerstone. So maybe you could source this env file only once,
just before setting up your profiles, and capture all the relevant
bits in your profile this way.

Just some ideas, i don't know how practical this would be.

I haven't use Yocto in ages, could you copy/paste your env file?

Chris
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

Re: [Qbs] Using ConanfileProbe with yocto cross compiling toolchain

[Richard Weickelt]

> I just stumbled across a scenario in which we use with multiple profiles for
> different cross compiling toolchains. Yocto creates a script which exports
> environment variables. We normally just extract the needed values from the
> script and set them as additional qbs profile settings.
> This leads to the problem that the ConanfileProbe just calls "conan
> install... -p..." without the needed environment variables. 
> Thus when
> install libraries they are getting compiled with the wrong toolchain/settings.
> What is the proper way of doing this? How do I set the
> environment variables? Should I set the variables in the Kit?

That would be one option, but sounds a bit inconvenient. If the environment
variables are only relevant when running the ConanfileProbe, then you would
need to:

1. Add an additionalEnvironmentVariables property to ConanfileProbe
2. Append these to the process environment in ConanfileProbe. See
   https://doc.qt.io/qbs/jsextension-process.html#setenv
3. Add an additional Probe item in your project file that extracts
   the environment variables from the script.
   Probes are currently executed in the order of their appearance in the qbs
   file, so make sure it comes first.

Would that be a solution for you? Then I would warmthly welcome a patch.

Note that instead of modifying qbs profile settings globally with your yocto
variables, you should be able to also create an intermediate Profile item in
your project files this way:

Project {
YoctoScriptProbe {
id: yocto
}
// This profile amends the profile set by Qt Creator or on the
// command line
Profile {
name: "yocto-profile"
baseProfile: project.profile
someModule.someProperty: yocto.someExtractedValue
}

// Every product now needs to specify the amended profile
// explicitly
Product {
profile: "yocto-profile"
}
}

Hope that helps.

Richard
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs

[Qbs] Using ConanfileProbe with yocto cross compiling toolchain

[Kai Dohmen via Qbs]

Hello,

I just stumbled across a scenario in which we use with multiple profiles
for different cross compiling toolchains. Yocto creates a script which
exports environment variables. We normally just extract the needed values
from the script and set them as additional qbs profile settings.
This leads to the problem that the ConanfileProbe just calls "conan
install... -p..." without the needed environment variables. Thus when
install libraries they are getting compiled with the wrong
toolchain/settings.
What is the proper way of doing this? How do I set the
environment variables? Should I set the variables in the Kit?

Thanks,
Kai

PS: the relevant conan documentation entry
https://docs.conan.io/en/latest/integrations/cross_platform/yocto.html#cross-building-conan-packages-with-the-sdk-toolchain
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs