Re: [Qbs] Using ConanfileProbe with yocto cross compiling toolchain
Hello.
I archived my goal with the following code:
Yocto_Conan_Qbs.qbs:
import qbs
import qbs.File
import qbs.FileInfo
import qbs.Probes
import qbs.Process
import qbs.TextFile
import qbs.Utilities
Project {
references: conanFileProbe.generatedFilesPath + "/conanbuildinfo.qbs"
Probe {
id: yoctoEnvProbe
/* in */
property path environmentSetupScriptPath:
"/home/psy-kai/Develop/Yocto_SDKs/armv7hf_reference/environment-setup-armv7vet2hf-neon-poky-linux-gnueabi"
/* out */
property var environmentVariables
/* private */
readonly property path _projectBuildDirectory:
project.buildDirectory
readonly property path _envExportScript:
FileInfo.cleanPath(FileInfo.joinPaths(project.sourceDirectory,
"env_export.sh"))
configure: {
var extractEnvironmentVariables = function(textFile){
var extractVariable = function(line){
var eqPos = line.indexOf("=");
var name = line.substr(0, eqPos);
var value = line.substr(eqPos+1);
if (!name)
throw "Invalid line in "+textFile.filePath();
return {"name": name, "value": value};
}
var envVars = [];
while (!textFile.atEof())
envVars.push(extractVariable(textFile.readLine()));
textFile.close();
return envVars;
}
var outputEnvFiles = FileInfo.cleanPath(
FileInfo.joinPaths(_projectBuildDirectory,
"yoctoenv",
Utilities.getHash(environmentSetupScriptPath)));
File.makePath(outputEnvFiles);
var emptyEnvFilePath = FileInfo.joinPaths(outputEnvFiles,
"emptyEnv");
var yoctoEnvFilePath = FileInfo.joinPaths(outputEnvFiles,
"yoctoEnv");
var process = new Process();
try {
process.exec("env",
["-", _envExportScript,
environmentSetupScriptPath, emptyEnvFilePath,
yoctoEnvFilePath],
true);
} finally {
process.close();
}
var emptyEnvVars = extractEnvironmentVariables(new
TextFile(emptyEnvFilePath));
var yoctoEnvVars = extractEnvironmentVariables(new
TextFile(yoctoEnvFilePath));
environmentVariables =
yoctoEnvVars.filter(function(yoctoEnvVar){
for (var i in emptyEnvVars) {
if (emptyEnvVars[i].name === yoctoEnvVar.name)
return false;
}
return true;
});
found = true;
}
}
Probes.ConanfileProbe {
id: conanFileProbe
conanfilePath: project.sourceDirectory+"/conanfile.txt"
additionalArguments: {
var args = ["-pr:b=imx6", "-b=missing"];
for (var i in yoctoEnvProbe.environmentVariables) {
var envVar = yoctoEnvProbe.environmentVariables[i];
args.push("-e:b="+envVar.name+"="+envVar.value);
}
return args
}
generators: "qbs"
}
}
env_export.sh:
#!/bin/sh
env > $2
source $1
env > $3
But this code has some restrictions (like using bash or zsh but not fish).
Thus I don't think that it should be made into the official Qbs codebase.
Maybe this code could be used in some How-To section or something.
If you really want me to contribute this as a "YoctoEnvironmentProbe",
despite the restrictions, I will do some cleanup an contribute it.
Kai
Am Fr., 15. Mai 2020 um 13:50 Uhr schrieb Richard Weickelt <
rich...@weickelt.de>:
>
> > Sorry. I meant how to extract the environment variables from the yocto
> script.
> > My idea was to source the script in a process
> > with https://doc.qt.io/qbs/jsextension-process.html
> > and then just query all the process environment variables with
> > https://doc.qt.io/qbs/jsextension-environment.html#currentenv.
> > This way I would not need to parse the yocto script myself in the
> > "YoctoEnvProbe".
>
> Maybe you can write another script that
> 1. prints the environment variables
> 2. then sources the yocto script
> 3. then prints the environment variables again
>
> Run this script in a Probe (with an appropriate shell) and parse the output
> somehow. You may also omit step 1 and just use all environment variables
> printed by script.
>
> Richard
> ___
> Qbs mailing list
> Qbs@qt-project.org
> https://lists.qt-project.org/listinfo/qbs
>
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] Using ConanfileProbe with yocto cross compiling toolchain
> > Do you know how I can easily extract the environment variables? > > https://doc.qt.io/qbs/jsextension-environment.html#currentenv seems like a > > solution but I don't know how to access a process environment with this. > > Which environment variables? If I understood you correctly, then you have > environment variables generated by yocto in a file/script which you need to > parse somehow in a probe. There is no probe for that, you would have to > implement your own. Sorry. I meant how to extract the environment variables from the yocto script. My idea was to source the script in a process with https://doc.qt.io/qbs/jsextension-process.html and then just query all the process environment variables with https://doc.qt.io/qbs/jsextension-environment.html#currentenv. This way I would not need to parse the yocto script myself in the "YoctoEnvProbe". ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
[Qbs] Using ConanfileProbe with yocto cross compiling toolchain
Hello, I just stumbled across a scenario in which we use with multiple profiles for different cross compiling toolchains. Yocto creates a script which exports environment variables. We normally just extract the needed values from the script and set them as additional qbs profile settings. This leads to the problem that the ConanfileProbe just calls "conan install... -p..." without the needed environment variables. Thus when install libraries they are getting compiled with the wrong toolchain/settings. What is the proper way of doing this? How do I set the environment variables? Should I set the variables in the Kit? Thanks, Kai PS: the relevant conan documentation entry https://docs.conan.io/en/latest/integrations/cross_platform/yocto.html#cross-building-conan-packages-with-the-sdk-toolchain ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] Using ConanfileProbe with yocto cross compiling toolchain
> 1. Add an additionalEnvironmentVariables property to ConanfileProbe > 2. Append these to the process environment in ConanfileProbe. See >https://doc.qt.io/qbs/jsextension-process.html#setenv > 3. Add an additional Probe item in your project file that extracts >the environment variables from the script. > > Probes are currently executed in the order of their appearance in the qbs > file, so make sure it comes first. > > Would that be a solution for you? Then I would warmthly welcome a patch. I will try this approach. But maybe I use `conan install -e` instead of setting the environment variables for the complete process. Do you know how I can easily extract the environment variables? https://doc.qt.io/qbs/jsextension-environment.html#currentenv seems like a solution but I don't know how to access a process environment with this. Thanks for the help so far. Kai ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
[Qbs] Building Qbs for conan/bintray on Gitlab
Hello, I try to build a Qbs conan package (with static linked Qt) so conan could just install Qbs with a simple "build_requires". Building the conan package works fine on my local machine. But when I use Gitlab-CI to build the packages it fails to build. To build locally I use the following Dockerfile: FROM conanio/gcc8 ENV CONAN_USERNAME="psy-kai" \ CONAN_CHANNEL="testing" \ CONAN_GCC_VERSIONS="8" \ CONAN_REMOTES=" https://api.bintray.com/conan/bincrafters/public-conan@True@bincrafters; RUN sudo pip install --upgrade conan_package_tools && \ conan user COPY *.py . COPY test_package test_package RUN python build.py Placing this into the cloned repo and running "docker build" works just fine. The complete build output of the crashing gitlab-ci build can be found at https://gitlab.com/Psy-Kai/conan-qbs/-/jobs/480352993. Hopefully anyone can help me with this issue. I think a pre-build Qbs conan package would help a lot, especially when using CI systems. Additionally some off topic: the documentation for the ConanfileProbe does not match with the current implementation. Maybe you should fix the docs before 1.16 release ;) Best Regards, Kai Dohmen ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] Building Qbs for conan/bintray on Gitlab
> I don't see compiler/linker invocations for any of the plugins in the gitlab > output, so it seems to be a build system issue. When running locally in a > Docker container, Qbs builds correctly. I have put above lines (without > invoking python build.py) in a Dockerfile and ran > > docker build > > Then I started a container with > > docker run -it --rm -v $PWD:/builds -w /builds qbsconan > > and invoked the build with > > docker exec CONTAINER_ID python build.py > > I guess this is a bit closer to what gitlab is doing, but maybe not 100%. So > what could be the difference? Oh. Interesting. I did not saw that the plugins are not compiled. But I don't know why. If you look into conanfile.py it just calls qmake -r qbs/qbs.pro CONFIG+=qbs_no_dev_install CONFIG+=qbs_no_man_install CONFIG+=qbs_use_bundled_qtscript Followed by a simple make -j1 // tools.cpu_count() will return 1 since every container just get 1 core assigned by gitlab.com gitlab-ci So something has to be wrong with the generated Makefiles? Should I try to archive the generated Makefiles for further analysis? > But if you manage to build Qbs from source in a Conan recipe using > bincrafters' Qt package, even better. Would you contribute and maintain a > Qbs Conan package? Yes, I would contribute maintain a Qbs Conan package. But if it should be an "official" Qbs Conan Package I would like to get some constraints like: where to host the packaging code; how to name the bintray etc. > In an open source project where almost all participants invest their spare > time to create something useful, it is good practice to not let others > guess. Would you mind sharing your findings? You may file a bug report or > submit a fix right away. Yeah, sorry for that. I opened a bug report. Sorry. Best Regards, Kai Dohmen Am Mo., 23. März 2020 um 11:01 Uhr schrieb Richard Weickelt < rich...@weickelt.de>: > Hello, > > > I try to build a Qbs conan package (with static linked Qt) so conan could > > just install Qbs with a simple "build_requires". Building the conan > package > > works fine on my local machine. But when I use Gitlab-CI to build the > > packages it fails to build. > > To build locally I use the following Dockerfile: > > > > FROM conanio/gcc8 > > > > ENV CONAN_USERNAME="psy-kai" \ > > CONAN_CHANNEL="testing" \ > > CONAN_GCC_VERSIONS="8" \ > > > > CONAN_REMOTES=" > https://api.bintray.com/conan/bincrafters/public-conan@True@bincrafters; > > > > RUN sudo pip install --upgrade conan_package_tools && \ > > conan user > > > > COPY *.py . > > COPY test_package test_package > > RUN python build.py > > > > Placing this into the cloned repo and running "docker build" works just > fine. > > The complete build output of the crashing gitlab-ci build can be found > > at https://gitlab.com/Psy-Kai/conan-qbs/-/jobs/480352993. > > I don't see compiler/linker invocations for any of the plugins in the > gitlab > output, so it seems to be a build system issue. When running locally in a > Docker container, Qbs builds correctly. I have put above lines (without > invoking python build.py) in a Dockerfile and ran > > docker build > > Then I started a container with > > docker run -it --rm -v $PWD:/builds -w /builds qbsconan > > and invoked the build with > > docker exec CONTAINER_ID python build.py > > I guess this is a bit closer to what gitlab is doing, but maybe not 100%. > So > what could be the difference? > > > I think a pre-build Qbs > > conan package would help a lot, especially when using CI systems. > > Yes, a Qbs Conan package in the bincrafters repo or in > https://conan.io/center/ would be great. I thought about just packaging > prebuilt binaries containing bundled Qt libraries as a first step for the > 1.16 release. The next Qbs release could then be built from source using > the > Qbs Conan package. > > But if you manage to build Qbs from source in a Conan recipe using > bincrafters' Qt package, even better. Would you contribute and maintain a > Qbs Conan package? > > > Additionally some off topic: the documentation for the ConanfileProbe > > does not match with the current implementation. Maybe you should fix the > > docs before 1.16 release ;) > > In an open source project where almost all participants invest their spare > time to create something useful, it is good practice to not let others > guess. Would you mind sharing your findings? You may file a bug report or > submit a fix right away. > > Thanks > Richard > ___ > Qbs mailing list > Qbs@qt-project.org > https://lists.qt-project.org/listinfo/qbs > ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
[Qbs] Why was protobuf.outputDir removed
Hello, the subject states my Question. Normally I create a static library from proto files. This static library can be linked into different applications. This really comes in handy when using conan. But since protobuf.outputDir was removed I cannot set the Exported includePaths dir properly. Furthermore when installing the generated header files I cannot use the Groups property installSourceBase to match the installed directory structure to the proto files ones. What is the right way of using proto files as a library? Should they get compiled or should they append to the depending products source files? Thanks, Kai ___ Qbs mailing list Qbs@qt-project.org https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] Why was protobuf.outputDir removed
Hello Ivan,
the use-case of installation arises from encapsulating the proto files into
a single library. I do this because then it is easier for me to use the
proto-stuff in different applications.
Since I started to use conan to track dependencies I needed to compile and
install the static library for use in the consuming application.
I would rather append the proto files to an products source files If there
is a (non hacky) way to it (with conan),
Kai
Am Di., 15. Sept. 2020 um 09:41 Uhr schrieb Иван Комиссаров <
abba...@gmail.com>:
> Hello, Kai
>
> First of all, the property was not removed but was made «private» and is
> now called _outputDir. You can still override it or read it.
>
> The reason why it was removed is that I didn’t see valid use-cases for
> that.
>
> For example, to export include paths you should do something like that:
>
> StaticLibrary {
> files: [«a.cpp», «b.proto»]
> Depends { name: «protobuf.cpp» }
> Export {
> Depends { name: «protobuf.cpp» } // this should export
> protobuf.cpp properties, including cpp.IncludePath
> }
> }
>
> However, your second use-case (installation) is totally new to me and it
> seems that you need to read this property.
> Unless someone can suggest a better solution, feel free to upload a patch
> to return the property.
> You might also want to assign custom tags for generated files (e.g.
> «proto_hpp», «proto_cpp») to make filtering more fine-grained and avoid
> overlapping with other headers.
>
> Ivan
>
> > 15 сент. 2020 г., в 09:00, Kai Dohmen via Qbs
> написал(а):
> >
> > Hello,
> >
> > the subject states my Question.
> > Normally I create a static library from proto files. This static library
> can be linked into different applications. This really comes in handy when
> using conan.
> > But since protobuf.outputDir was removed I cannot set the Exported
> includePaths dir properly.
> > Furthermore when installing the generated header files I cannot use the
> Groups property installSourceBase to match the installed directory
> structure to the proto files ones.
> >
> > What is the right way of using proto files as a library? Should they get
> compiled or should they append to the depending products source files?
> >
> > Thanks,
> > Kai
> > ___
> > Qbs mailing list
> > Qbs@qt-project.org
> > https://lists.qt-project.org/listinfo/qbs
>
>
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] Problem getting build order right when I use a header generator
Hello,
did you try to append "hpp" to the static libraries type property?
Maybe this would force Qbs to generate the headers before consuming the
library.
Regards, Kai
Am Mi., 13. Juli 2022 um 22:50 Uhr schrieb Jochen Becher <
jochen_bec...@gmx.de>:
> Another observation:
>
> After I built the project once with one job at a time and it succeeded,
> it also succeeds after "Clean" and building with 12 parallel jobs. So,
> once the dependency between the source file and the header file is
> established, it gets not lost until you delete the .bg file.
>
> As it works within the static library but just not when an external
> source file depends on that generated header file, I assume it is an
> issue in qbs.
>
> Any comments?
>
>
> Am Dienstag, dem 12.07.2022 um 22:08 +0200 schrieb Jochen Becher:
> > One more information:
> >
> > If I set the number of parallel jobs to 1 then it works fine. The
> > order
> > of the build tasks seems to be correct. Just the compilation of
> > source
> > files outside of the library but depending on a header file created
> > from the library seems not to wait until the header files are done.
> > Of
> > course, this obviously works with linking of object files and static
> > libraries. It looks like an issue in qbs to me.
> >
> > Can someone please comment on this before I create a minimal example
> > and issue?
> >
> > Regards, Jochen
> >
> >
> > Am Samstag, dem 09.07.2022 um 22:56 +0200 schrieb Jochen Becher:
> > > Hi,
> > >
> > > I have the following problem where I could not find a solution
> > > myself:
> > >
> > > I created a new module which uses a code generator "compiler-
> > > generator"
> > > that I wrote to create header and source files from input files. I
> > > used
> > > the lexyacc module as a template. (see end of mail for module
> > > source)
> > >
> > > I use that module in a product that creates a static library, like
> > > that:
> > >
> > > StaticLibrary {
> > > name: "corelib"
> > >
> > > Export {
> > > Depends { name: "cpp" }
> > > cpp.includePaths: ["src",
> > > exportingProduct.buildDirectory]
> > > }
> > >
> > > Depends { name: "compgen" }
> > > Depends { name: "compiler-generator" }
> > >
> > > cpp.includePaths: [ "src", product.buildDirectory ]
> > >
> > > Group {
> > > name: "frontend-header-generator"
> > > fileTags: "compgen.input"
> > > compgen.outputTag: "hpp"
> > > files: [
> > > "src/frontend/tokens.h.in",
> > > ]
> > > }
> > >
> > > Group {
> > > name: "frontend-source-generator"
> > > fileTags: "compgen.input"
> > > compgen.outputTag: "c"
> > > files: [
> > > "src/frontend/lexer.c.in",
> > > ]
> > > }
> > >
> > > Group {
> > > name: "frontend"
> > > files: [
> > > "src/frontend/lexer.h",
> > > "src/frontend/parser.c",
> > > "src/frontend/parser.h",
> > > ]
> > > }
> > >
> > > Building that library works fine.
> > >
> > > But some other tools also want to use the created header file
> > > "tokens.h". These applications (e.g., "compiler") depends on
> > > "corelib".
> > >
> > > But the application source is compiled before the header file
> > > "tokens.h" is generated from "tokens.h.in" and so it is missing.
> > >
> > > What is the best way to enforce that the dependency on corelib from
> > > an
> > > application first builds the library corelib with the generated
> > > header
> > > and source files and THEN builds the dependend source files?
> > >
> > > Regards, Jochen
> > >
> > >
> > > --- the generator module "compgen.qbs" ---
> > >
> > > import qbs
> > > import qbs.FileInfo
> > >
> > > Module {
> > >
> > > Depends { name: "cpp" }
> > >
> > > property string outputTag: "c"
> > > readonly property string outputDir: product.buildDirectory
> > >
> > > Rule {
> > > inputs: [ "compgen.input" ]
> > > outputFileTags: [ product.compgen.outputTag ]
> > > explicitlyDependsOnFromDependencies: [ "compiler-generator"
> > > ]
> > >
> > > Artifact {
> > > filePath: input.completeBaseName
> > > fileTags: [ product.compgen.outputTag ]
> > > cpp.includePaths: [].concat(input.cpp.includePaths,
> > > FileInfo.path(input.filePath), input.compgen.outputDir)
> > > }
> > >
> > > prepare: {
> > > var command = explicitlyDependsOn["compiler-
> > > generator"][0].filePath
> > > var arguments = [
> > > "-o", output.filePath,
> > > input.filePath
> > > ]
> > > var cmd = new Command(command, arguments)
> > > cmd.description = "generating " + output.filePath + "
> > > from
> > > " + input.filePath
> > >
