Re: [Qbs] Interfacing Qbs to Conan
> It would be great if Qbs 1.16 could offer a complete Conan story. We have
> roughly a month until feature freeze.
The ticket(s) I will create are not a requirement for the Conan ModuleProvider
we implemented. So we should not have a problem with this deadline.
The things from the ticket(s) would just allow a simpler/better implementation
of the ModuleProvider and improve its usability.
> It might be beneficial if module providers [...]
Yeah. Those three things would really be helpful.
> I do not think they should have direct access to probes/properties defined in
> the project/product by the user because this would be an "invisible"
> dependency. That means: A provider should not rely on
> project.mycustomProperty.
Okay. That's true. And if the other three things would be possible, it would be
unnecessary to have access to the probes/properties defined in the
project/product.
> Are you aware that module providers operate on product level? You can
> (currently) configure them in the profile and in the product.
Yeah but that's not correct for the Conan ModuleProvider and probably also for
other ModuleProviders. What they need are properties on *project* level.
They should not be configured in a profile because they are specific to a
project. And you don't want to repeat them for every product in the project.
> We could also add the capability to "configure" the fallback provider via the
> Depends item [...]
I also thought about this. However, I would call the property simply
"providers" instead of "fallbackProviders":
Depends {
name: "mylib"
providers: [
"conan",
"pkgConfig"
]
}
>https://codereview.qt-project.org/c/qbs/qbs/+/288927 achieves automatic
> probe re-execution by assigning the modification time stamp of the conanfile
> to a property of the probe.
Yeah. Something like this. Except for ModuleProviders.
>An --force-module-generator-execution command line flag might be worth to
> add to qbs resolve.
I would call it --force-module-provider-execution to make it clear that its
related to the ModuleProviders.
So to summarize: we have the following new requirements regarding
ModuleProviders:
1. ModuleProviders should be able to define their own probes.
2. ModuleProviders should be able to define a logic to "invalidate" their
generated modules (force regeneration of the modules).
3. ModuleProviders should be able to access the project and product they live
in.
4. Projects should be able to configure ModuleProviders like it is possible in
products.
5. Depends items should be able to define which ModuleProviders to use. Doing
this invokes the ModuleProvider like a fallback ModuleProvider (meaning it gets
the complete "name" of the Depends item).
6. qbs resolve should provide a command line flag
--force-module-provider-execution to force regeneration of modules by
ModuleProviders.
Should I create one or multiple tickets for these requirements?
To me it sounds like 1, 3 and 4 might be related while the others are rather
independent.
Best
Jochen
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] Interfacing Qbs to Conan
Jochen,
> Basically, we are doing it the way you described. However, we decided not
> to make a conan module which the generated modules depend on because we
> expect that this would lead to multiple invokations of `conan install`
> (one for each generated module) which we want to avoid.
This should not be the case. Probes are cached by the location of their
definition and the input properties (snapshot before running configure).
> Yeah. While working on the Conan ModuleProvider, we encountered a few
> obstactles. Luckily, it seems we are able to work around them. I list
> these obstacles here along with the workarounds we are currently
> implementing so we can discuss them. I would then create a feature
> request ticket to extend the ModuleProvider accordingly.
It would be great if Qbs 1.16 could offer a complete Conan story. We have
roughly a month until feature freeze.
> So the following things might be helpful when writing a ModuleProvider:
>
> 1.) Access to (project-level) probes It is quite likely that a
> ModuleProvider needs some external tools or files and those should be
> detected using probes. In our case, we need to detect the conan
> executable and the conanfile(s). Our workaround is to let the
> project-level probe execute `conan install` so the ModuleProvider does
> not need to know the location of the conan executable and the
> conanfile(s). This then produces a conanbuildinfo.json which the probe
> places in a defined location inside the build directory so that the
> ModuleProvider is able to find it.
It might be beneficial if module providers:
a. could define their own probe items.
b. have access to the project and the product they live in.
c. could be set up in projects like it is possible in products
I do not think they should have direct access to probes/properties defined
in the project/product by the user because this would be an "invisible"
dependency. That means: A provider should not rely on project.mycustomProperty.
> 2.) Access to the project the ModuleProvider is running in A
> ModuleProvider might need or save some files in the buildDirectory. So
> access to the project.buildDirectory and project.sourceDirectory would be
> helpful. In our case, we need to locate the conanbuildinfo.json in the
> buildDirectory. Enhancement 1.) might also solve this. However, in our
> project, we might end up with several conanbuildinfo.json files
> (hierarchy of projects) and therefore, access to project.name to
> differentiate the files might be helpful. Our workaround to get the
> buildDirectory is to assume that the outputBaseDir of the ModuleProvider
> is inside the build directory. So we simply do something like `var
> buildDirectory = FileInfo.cleanPath( outputBaseDir + '/../..' )`. For the
> multiple conanbuildinfo.json files issue, we simply take all of them
> inside a defined directory.
You can define custom properties for your module provider.
Are you aware that module providers operate on product level? You can
(currently) configure them in the profile and in the product. Would the
following approach do what you want?
// module-providers/conan/provider.qbs
// Sets up a module for each key in dependencies
ModuleProvider {
property var dependencies
// ...
}
Project {
property var dependencies: conan.dependencies
// https://codereview.qt-project.org/c/qbs/qbs/+/288927
Probes.ConanfileProbe {
id: conan
conanfilePath: "conanfile.py"
}
Product {
Depends { name: "conan.somelib" }
moduleProviders.conan.dependencies: project.dependencies
}
}
> 3.) Access to the full name of the Depends items for which the
> ModuleProvider was called This works for the fallback ModuleProvider
> AFAIK. However, at the moment there can be only one fallback
> ModuleProvider AFAIK. So it would not be easily possible for a project to
> use both pkgConfig and Conan, right? Allowing access to the full name of
> the Depends item in a regular ModuelProvider when evaluating the
> relativeSearchPaths property would allow writing something like `Depends
> { name: "conan.somePackage" }` which would invoke the conan
> ModuleProvider which would then generate the Module for "somePackage".
> Our workaround is to generate Modules for all dependencies from the
> conanfile, no matter if they are used as Depends items in Qbs or not. So
> this is not a real issue in our case but it limits ModuleProviders which
> do not have something like the conanfile but only rely on the name of the
> dependency.
True. There should be a way to select fallback providers. At least you can
override the fallback provider by putting your own provider into a
somePath/module-providers/__fallback folder and add somePath to qbsSearchPaths.
I wonder how a better and generic solution would look like. We could add a
moduleProviders.defaultFallbackProvider configuration option or we could
replace Depends.enableFallback by Depends.providers and then explicitly
specify a
Re: [Qbs] Interfacing Qbs to Conan
Hi Richard!
> I hope this gives you some ideas how to move forward
Thanks Richard! We really appreciate your help!
> There might be different solutions to this, but what I think would be simple
> and straight forward
Basically, we are doing it the way you described.
However, we decided not to make a conan module which the generated modules
depend on because we expect that this would lead to multiple invokations of
`conan install` (one for each generated module) which we want to avoid.
So we decided to have a project-level probe instead which detects the conan
executable and the conanfile and runs `conan install`.
This way, one can also easily define the path to the conanfile inside the Qbs
files.
> I would be more than happy to discuss any issues appearing on the way.
Yeah. While working on the Conan ModuleProvider, we encountered a few
obstactles. Luckily, it seems we are able to work around them.
I list these obstacles here along with the workarounds we are currently
implementing so we can discuss them.
I would then create a feature request ticket to extend the ModuleProvider
accordingly.
So the following things might be helpful when writing a ModuleProvider:
1.) Access to (project-level) probes
It is quite likely that a ModuleProvider needs some external tools or files and
those should be detected using probes.
In our case, we need to detect the conan executable and the conanfile(s).
Our workaround is to let the project-level probe execute `conan install` so the
ModuleProvider does not need to know the location of the conan executable and
the conanfile(s). This then produces a conanbuildinfo.json which the probe
places in a defined location inside the build directory so that the
ModuleProvider is able to find it.
2.) Access to the project the ModuleProvider is running in
A ModuleProvider might need or save some files in the buildDirectory. So access
to the project.buildDirectory and project.sourceDirectory would be helpful.
In our case, we need to locate the conanbuildinfo.json in the buildDirectory.
Enhancement 1.) might also solve this. However, in our project, we might end up
with several conanbuildinfo.json files (hierarchy of projects) and therefore,
access to project.name to differentiate the files might be helpful.
Our workaround to get the buildDirectory is to assume that the outputBaseDir of
the ModuleProvider is inside the build directory. So we simply do something
like `var buildDirectory = FileInfo.cleanPath( outputBaseDir + '/../..' )`.
For the multiple conanbuildinfo.json files issue, we simply take all of them
inside a defined directory.
3.) Access to the full name of the Depends items for which the ModuleProvider
was called
This works for the fallback ModuleProvider AFAIK. However, at the moment there
can be only one fallback ModuleProvider AFAIK. So it would not be easily
possible for a project to use both pkgConfig and Conan, right?
Allowing access to the full name of the Depends item in a regular
ModuelProvider when evaluating the relativeSearchPaths property would allow
writing something like `Depends { name: "conan.somePackage" }` which would
invoke the conan ModuleProvider which would then generate the Module for
"somePackage".
Our workaround is to generate Modules for all dependencies from the conanfile,
no matter if they are used as Depends items in Qbs or not.
So this is not a real issue in our case but it limits ModuleProviders which do
not have something like the conanfile but only rely on the name of the
dependency.
4.) Mechanism to force execution of the ModuleProvider
Since ModuleProvider generate modules on a dynamic basis, it is not unlikely
that the environment changes and the ModuleProvider needs to re-run. At the
moment, this can be forced with the --force-probe-execution command line
parameter. However, it would be nice if Qbs could detects this automatically.
I'm not sure how this could be realized. Maybe a property on the ModuleProvider
that, when it evaluates to `true`, forces re-execution of the ModuleProvider?
In our case, we would like to re-run the ModuleProvider when the conanfile (or
conanbuildinfo.json) has changed.
Our workaround is that we have to call qbs with --force-probe-execution
manually when the conanfile has changed.
So would those enhancements make sense?
Best regards
Jochen
___
Qbs mailing list
Qbs@qt-project.org
https://lists.qt-project.org/listinfo/qbs
Re: [Qbs] Interfacing Qbs to Conan
Hello Leon,
I am really glad to read that Clausmark is looking into interfacing Qbs with
Conan. Since Conan gets more and more traction and brings great advantages,
Qbs should support it well. That's why I want to share some more detailed
thoughts on this topic. I'll start with what I originally had in mind and
later get to module providers.
> Here's the probe. I is instantiated at project level where the value of
> 'generatedQbsFilePath' is added to the 'references' property. If you run
> it like this, 'targetArch' will be undefined.
I assume that this has already been sufficiently answered and since I was
told that you are now trying to implemented a conan module provider, I will
not comment on it. However, I had this topic on my personal backlog for a
while and came up with the following probe (WIP):
https://codereview.qt-project.org/c/qbs/qbs/+/288927
You may find this useful and I would be glad to get feedback.
> How would I access a property set in a profile? I think I've tried it
> before but I probably did something wrong.
Module properties set in profiles are only accessible from within products
and modules.
> Also in the blog post about the release of qbs 1.15.0 it says the
> following: Probes on the project level are now evaluated before Profile
> items. That way a Qbs project can easily interface to package managers
> like Conan or vcpkg and resolve all build dependencies including compiler
> toolchains in a clean and platform-agnostic way. [...]
As Christian already wrote, this is about Profile items. My assumption is
the following:
1. When using Conan, you want to package up _everything_ necessary to build
your project. Not only the relevant libaries, but also toolchains. You want
to have the same tools on every computer as well as in your CI system and
you might have different host operating systems.
2. You have modules for all relevant libraries and your only problem is the
connection from your conanfile to setting up these modules.
3. You do not want to or cannot use conan's qbs generator because your
packages are missing important meta information or do not fit into conan's
standard layout. Conan't built-in qbs generator is very inflexible.
Thus, I thought users could write a simple Probe to run conan install and
extract all relevant meta information from the resulting
conanbuildinfo.json. Since Probe items are now executed before Profile items
are evaluated, users could do something like:
Project {
Probes.ConanfileProbe {
id: conan
conanfilePath: path + "/conanfile.py"
options: ({someOpt: "bla"})
}
Profile {
name: "gcc"
mylib.includePaths: conan.dependencies["mylib"].include_paths
qbs.toolchain: "gcc"
cpp.installPath: conan.dependencies["gcc-arm-none-eabi"].rootpath +
"/bin"
}
}
This way you could set up a completely self-containing project and you would
never again mess around with manual dependency installation on your computer
(toolchains included). Note how simple, straight-forward and
easy-to-understand this is compared to how CMake interacts with Conan.
However, there is unfortunately a limitation: Qt Creator is not able to use
profiles with a custom name specified in a project. Qt Creator can only
invoke builds with its own cryptic profile naming scheme (for instance
qtc_Desktop__9d3c5e00). One possible, but ugly work-around for this problem
is to create a dummy profile that shadows an existing profile in Qt Creator.
Like this:
Profile {
name: "qtc_Desktop__9d3c5e00"
baseProfile: "gcc"
}
Qbs will then no longer use the values from Qt Creator's own profiles, but
the ones set in the Profile item in the project. That means, developer
participating in the project would need to add such a dummy Profile item to
the project which is not really what we want.
I think it would be a great improvement in QtCreator to extract Qbs' Profile
items as read-only kits when opening a Qbs project. And it would be multiple
times easier than anything existing today.
> Doesn't that mean that profile specific properties are inaccessible at
> the time the probe is evaluated?
True (on project-level). The "qbs" module is a bit special because it is
loaded very early and is always implicitly available everywhere. The
qbs.architecture property is another oddity because if it is not set by the
profile, it will be set later by the cpp module.
> How would I reference the qbs file generated by conan though? I'm only
> doing it with this project level probe because I thought there was no
> other way, or is there?
If you want to reference the qbs file created by conan's qbs generator, then
yes, would need a probe on Project level invoking conan install
/path/to/conanfile -g qbs ... The qbs file should be generated somewhere
under
