Re: [rtems-source-builder commit] sb/setbuilder: Correctly create build set tar files
On 29/9/22 6:01 pm, Christian MAUDERER wrote:
> Am 29.09.22 um 09:52 schrieb Chris Johns:
>>
>>
>> On 29/9/22 5:13 pm, Christian MAUDERER wrote:
>>> Am 29.09.22 um 08:56 schrieb Chris Johns:
On 29/9/2022 4:55 pm, Christian MAUDERER wrote:
> Am 29.09.22 um 08:54 schrieb Chris Johns:
>> On 29/9/2022 4:42 pm, Christian MAUDERER wrote:
It could be a bug if the tools builds work, ie 6/rtems-*. Please raise
a
ticket?
>>>
>>> The tool builds work except for the 6/rtems-microblaze.
>>
>> Thanks, I will take a look.
>>
>
> I just checked it: There is the same behavior for devel/dtc (which is much
> faster to build). Without the patch an archive is created. With the patch
> it
> doesn't create one.
Awesome, that will help.
>>>
>>> In case of dtc, it seems to not work because dtc is in
>>>
>>> devel/dtc-1.6.1-1.cfg
>>>
>>> which ends in a ".cfg" and not in a ".bset".
>>>
>>> The bset_tar(...) function is only called if the have_staging is set here:
>>>
>>> https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/setbuilder.py?id=161b7f108c3daa0f71ac289ba048a68b730d422c#n623
>>>
>>>
>>>
>>> That variable seems to be only set to True here:
>>>
>>> https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/setbuilder.py?id=161b7f108c3daa0f71ac289ba048a68b730d422c#n489
>>>
>>>
>>>
>>> And that statement is in a
>>>
>>> if configs[s].endswith('.bset')
>>>
>>> Does that mean that only bsets can be packed?
>>
>> It means the build is not being staged and so no bset tar generation. At a
>> guess
>> a single package in a build does not need to be staged so that step is
>> missed.
>>
>> Just a bug. If you could please raise a ticket and assign to me I will take
>> care
>> of this.
>>
>
> https://devel.rtems.org/ticket/4730#ticket
Thanks.
> Is there anything I can help with solving that?
I have a fix so some testing a bit would be a big help.
The staging logic was a progression of changes at the time and took some effort.
Once it was sorted some extra and unneeded complexity was left and that can be
remove. All bsets and cfgs should be staged so removing the logic that made only
nested builds stage fixes it for dtc. I just need to test the more complex
builds.
The microblaze is a separate gcc version so I am not sure about it. I will to
look into it next.
Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel
Re: [rtems-source-builder commit] sb/setbuilder: Correctly create build set tar files
Am 29.09.22 um 09:52 schrieb Chris Johns:
On 29/9/22 5:13 pm, Christian MAUDERER wrote:
Am 29.09.22 um 08:56 schrieb Chris Johns:
On 29/9/2022 4:55 pm, Christian MAUDERER wrote:
Am 29.09.22 um 08:54 schrieb Chris Johns:
On 29/9/2022 4:42 pm, Christian MAUDERER wrote:
It could be a bug if the tools builds work, ie 6/rtems-*. Please raise a
ticket?
The tool builds work except for the 6/rtems-microblaze.
Thanks, I will take a look.
I just checked it: There is the same behavior for devel/dtc (which is much
faster to build). Without the patch an archive is created. With the patch it
doesn't create one.
Awesome, that will help.
In case of dtc, it seems to not work because dtc is in
devel/dtc-1.6.1-1.cfg
which ends in a ".cfg" and not in a ".bset".
The bset_tar(...) function is only called if the have_staging is set here:
https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/setbuilder.py?id=161b7f108c3daa0f71ac289ba048a68b730d422c#n623
That variable seems to be only set to True here:
https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/setbuilder.py?id=161b7f108c3daa0f71ac289ba048a68b730d422c#n489
And that statement is in a
if configs[s].endswith('.bset')
Does that mean that only bsets can be packed?
It means the build is not being staged and so no bset tar generation. At a guess
a single package in a build does not need to be staged so that step is missed.
Just a bug. If you could please raise a ticket and assign to me I will take care
of this.
https://devel.rtems.org/ticket/4730#ticket
Is there anything I can help with solving that?
Best regards
Christian
Nice work getting it down to here. I appreciate it. >
I think having a tarball for tools like qemu or dtc could be quite useful too.
Agreed.
PS: Not sure yet why microblaze is affected too. That maybe is another case.
Thanks. Getting to it with rtems-deployment repo but I hit 5 verses devel branch
issues.
Chris
--
embedded brains GmbH
Herr Christian MAUDERER
Dornierstr. 4
82178 Puchheim
Germany
email: christian.maude...@embedded-brains.de
phone: +49-89-18 94 741 - 18
mobile: +49-176-152 206 08
Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel
Re: [rtems-source-builder commit] sb/setbuilder: Correctly create build set tar files
On 29/9/22 5:13 pm, Christian MAUDERER wrote:
> Am 29.09.22 um 08:56 schrieb Chris Johns:
>> On 29/9/2022 4:55 pm, Christian MAUDERER wrote:
>>> Am 29.09.22 um 08:54 schrieb Chris Johns:
On 29/9/2022 4:42 pm, Christian MAUDERER wrote:
>> It could be a bug if the tools builds work, ie 6/rtems-*. Please raise a
>> ticket?
>
> The tool builds work except for the 6/rtems-microblaze.
Thanks, I will take a look.
>>>
>>> I just checked it: There is the same behavior for devel/dtc (which is much
>>> faster to build). Without the patch an archive is created. With the patch it
>>> doesn't create one.
>>
>> Awesome, that will help.
>>
>
> In case of dtc, it seems to not work because dtc is in
>
> devel/dtc-1.6.1-1.cfg
>
> which ends in a ".cfg" and not in a ".bset".
>
> The bset_tar(...) function is only called if the have_staging is set here:
>
> https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/setbuilder.py?id=161b7f108c3daa0f71ac289ba048a68b730d422c#n623
>
>
> That variable seems to be only set to True here:
>
> https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/setbuilder.py?id=161b7f108c3daa0f71ac289ba048a68b730d422c#n489
>
>
> And that statement is in a
>
> if configs[s].endswith('.bset')
>
> Does that mean that only bsets can be packed?
It means the build is not being staged and so no bset tar generation. At a guess
a single package in a build does not need to be staged so that step is missed.
Just a bug. If you could please raise a ticket and assign to me I will take care
of this.
Nice work getting it down to here. I appreciate it.
> I think having a tarball for tools like qemu or dtc could be quite useful too.
Agreed.
> PS: Not sure yet why microblaze is affected too. That maybe is another case.
Thanks. Getting to it with rtems-deployment repo but I hit 5 verses devel branch
issues.
Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel
Re: [rtems-source-builder commit] sb/setbuilder: Correctly create build set tar files
Am 29.09.22 um 08:56 schrieb Chris Johns:
On 29/9/2022 4:55 pm, Christian MAUDERER wrote:
Am 29.09.22 um 08:54 schrieb Chris Johns:
On 29/9/2022 4:42 pm, Christian MAUDERER wrote:
It could be a bug if the tools builds work, ie 6/rtems-*. Please raise a
ticket?
The tool builds work except for the 6/rtems-microblaze.
Thanks, I will take a look.
I just checked it: There is the same behavior for devel/dtc (which is much
faster to build). Without the patch an archive is created. With the patch it
doesn't create one.
Awesome, that will help.
In case of dtc, it seems to not work because dtc is in
devel/dtc-1.6.1-1.cfg
which ends in a ".cfg" and not in a ".bset".
The bset_tar(...) function is only called if the have_staging is set here:
https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/setbuilder.py?id=161b7f108c3daa0f71ac289ba048a68b730d422c#n623
That variable seems to be only set to True here:
https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/setbuilder.py?id=161b7f108c3daa0f71ac289ba048a68b730d422c#n489
And that statement is in a
if configs[s].endswith('.bset')
Does that mean that only bsets can be packed? I think having a tarball
for tools like qemu or dtc could be quite useful too.
PS: Not sure yet why microblaze is affected too. That maybe is another case.
Best regards
Christian
--
embedded brains GmbH
Herr Christian MAUDERER
Dornierstr. 4
82178 Puchheim
Germany
email: christian.maude...@embedded-brains.de
phone: +49-89-18 94 741 - 18
mobile: +49-176-152 206 08
Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel
Re: [rtems-source-builder commit] sb/setbuilder: Correctly create build set tar files
On 29/9/2022 4:55 pm, Christian MAUDERER wrote: > Am 29.09.22 um 08:54 schrieb Chris Johns: >> On 29/9/2022 4:42 pm, Christian MAUDERER wrote: It could be a bug if the tools builds work, ie 6/rtems-*. Please raise a ticket? >>> >>> The tool builds work except for the 6/rtems-microblaze. >> >> Thanks, I will take a look. >> > > I just checked it: There is the same behavior for devel/dtc (which is much > faster to build). Without the patch an archive is created. With the patch it > doesn't create one. Awesome, that will help. Chris ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [rtems-source-builder commit] sb/setbuilder: Correctly create build set tar files
Am 29.09.22 um 08:54 schrieb Chris Johns: On 29/9/2022 4:42 pm, Christian MAUDERER wrote: It could be a bug if the tools builds work, ie 6/rtems-*. Please raise a ticket? The tool builds work except for the 6/rtems-microblaze. Thanks, I will take a look. I just checked it: There is the same behavior for devel/dtc (which is much faster to build). Without the patch an archive is created. With the patch it doesn't create one. Best regards Christian -- embedded brains GmbH Herr Christian MAUDERER Dornierstr. 4 82178 Puchheim Germany email: christian.maude...@embedded-brains.de phone: +49-89-18 94 741 - 18 mobile: +49-176-152 206 08 Registergericht: Amtsgericht München Registernummer: HRB 157899 Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler Unsere Datenschutzerklärung finden Sie hier: https://embedded-brains.de/datenschutzerklaerung/ ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [rtems-source-builder commit] sb/setbuilder: Correctly create build set tar files
On 29/9/2022 4:42 pm, Christian MAUDERER wrote: >> It could be a bug if the tools builds work, ie 6/rtems-*. Please raise a >> ticket? > > The tool builds work except for the 6/rtems-microblaze. Thanks, I will take a look. Chris ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [rtems-source-builder commit] sb/setbuilder: Correctly create build set tar files
Hello Chris, thanks for the response. Am 29.09.22 um 01:40 schrieb Chris Johns: On 28/9/2022 11:42 pm, Christian MAUDERER wrote: Hello, with this patch, I don't get a tar for devel/qemu and for the 6/rtems-microblaze anymore. All other 6/rtems-* toolchains work without problems. I haven't tested a lot of the other packages. The bset for microblaze is a bit different from the other ones. But I'm not yet sure what the relevant difference is. @Chris: With your change: What is necessary that a bset can generate a tar archive? It could be a bug if the tools builds work, ie 6/rtems-*. Please raise a ticket? The tool builds work except for the 6/rtems-microblaze. I changed the --bset-tar-file option to create a single tarfile of the final staged output of a build. There were a few commits to get this right so I assume you are testing on the latest? I used the current master during the tests. I reverted only the single "Correctly create build set tar files" commit to check whether it was the relevant one. Without this single commit but all others still in place the qemu tar file is created. Incremental tarfiles based on separate buildsets in a buildset may have worked in some cases however there were some basic issues in how it was implemented. When you add deployment requirements on top it did not match up well and it was confusing. The best solution was to rebase the tarfile against the final staged output as that is known to be correct in all cases. I'm not sure whether I understand that, but that is because I never analyzed how the source builder generates the tar files. I'll try to read a bit more documentation and sources. Best regards Christian I have created a https://git.rtems.org/chrisj/rtems-deployment.git repo and in it a config test directory (https://git.rtems.org/chrisj/rtems-deployment.git/tree/config/test). I will look at adding a test to make sure we catch any issues. Thanks Chris -- embedded brains GmbH Herr Christian MAUDERER Dornierstr. 4 82178 Puchheim Germany email: christian.maude...@embedded-brains.de phone: +49-89-18 94 741 - 18 mobile: +49-176-152 206 08 Registergericht: Amtsgericht München Registernummer: HRB 157899 Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler Unsere Datenschutzerklärung finden Sie hier: https://embedded-brains.de/datenschutzerklaerung/ ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [rtems-source-builder commit] sb/setbuilder: Correctly create build set tar files
On 28/9/2022 11:42 pm, Christian MAUDERER wrote: > Hello, > > with this patch, I don't get a tar for devel/qemu and for the > 6/rtems-microblaze > anymore. All other 6/rtems-* toolchains work without problems. I haven't > tested > a lot of the other packages. > > The bset for microblaze is a bit different from the other ones. But I'm not > yet > sure what the relevant difference is. > > @Chris: With your change: What is necessary that a bset can generate a tar > archive? It could be a bug if the tools builds work, ie 6/rtems-*. Please raise a ticket? I changed the --bset-tar-file option to create a single tarfile of the final staged output of a build. There were a few commits to get this right so I assume you are testing on the latest? Incremental tarfiles based on separate buildsets in a buildset may have worked in some cases however there were some basic issues in how it was implemented. When you add deployment requirements on top it did not match up well and it was confusing. The best solution was to rebase the tarfile against the final staged output as that is known to be correct in all cases. I have created a https://git.rtems.org/chrisj/rtems-deployment.git repo and in it a config test directory (https://git.rtems.org/chrisj/rtems-deployment.git/tree/config/test). I will look at adding a test to make sure we catch any issues. Thanks Chris ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [rtems-source-builder commit] sb/setbuilder: Correctly create build set tar files
Hello,
with this patch, I don't get a tar for devel/qemu and for the
6/rtems-microblaze anymore. All other 6/rtems-* toolchains work without
problems. I haven't tested a lot of the other packages.
The bset for microblaze is a bit different from the other ones. But I'm
not yet sure what the relevant difference is.
@Chris: With your change: What is necessary that a bset can generate a
tar archive?
Best regards
Christian
Am 19.09.22 um 23:24 schrieb Chris Johns:
Module:rtems-source-builder
Branch:5
Commit:828feec622c3ebfdf96a50f5789fcace93717f82
Changeset:
http://git.rtems.org/rtems-source-builder/commit/?id=828feec622c3ebfdf96a50f5789fcace93717f82
Author:Chris Johns
Date: Thu Sep 1 16:20:44 2022 +1000
sb/setbuilder: Correctly create build set tar files
- Make a single tarfile for all buildsets built
- Use the staging tree as the tarfile source
- Use python's tarfile module
- Create a config.file object without loading a .cfg file
Updates #4716
---
source-builder/sb/config.py | 5 ++--
source-builder/sb/setbuilder.py | 58 +
2 files changed, 39 insertions(+), 24 deletions(-)
diff --git a/source-builder/sb/config.py b/source-builder/sb/config.py
index 9250896..24590eb 100644
--- a/source-builder/sb/config.py
+++ b/source-builder/sb/config.py
@@ -258,7 +258,7 @@ class file:
re.compile('%select'),
re.compile('%disable') ]
-def __init__(self, name, opts, macros = None):
+def __init__(self, name, opts, macros = None, load = True):
log.trace('config: %s: initialising' % (name))
self.opts = opts
self.init_name = name
@@ -267,7 +267,8 @@ class file:
self.sf = re.compile(r'%\([^\)]+\)')
self.set_macros(macros)
self._reset(name)
-self.load(name)
+if load:
+self.load(name)
def __str__(self):
diff --git a/source-builder/sb/setbuilder.py b/source-builder/sb/setbuilder.py
index 55bd86f..36fca46 100644
--- a/source-builder/sb/setbuilder.py
+++ b/source-builder/sb/setbuilder.py
@@ -30,11 +30,14 @@ import glob
import operator
import os
import sys
+import tarfile
+
import textwrap
try:
import build
import check
+import config
import error
import log
import mailer
@@ -259,21 +262,31 @@ class buildset:
self.root_copy(_build.config.expand('%{buildroot}'),
_build.config.expand('%{_tmproot}'))
-def bset_tar(self, _build):
-tardir = _build.config.expand('%{_tardir}')
-if (self.opts.get_arg('--bset-tar-file') or
self.opts.canadian_cross()) \
- and not _build.macros.get('%{_disable_packaging}'):
+def bset_tar(self, stagingroot):
+if self.opts.get_arg('--bset-tar-file') or self.opts.canadian_cross():
+# Use a config to expand the macros because it supports all
+# expansions, ie %{_cwd}
+cfg = config.file(self.bset, self.opts, self.macros, load=False)
+prefix = cfg.expand('%{_prefix}')
+tardir = cfg.expand('%{_tardir}')
path.mkdir(tardir)
-tar = path.join(tardir,
-_build.config.expand('%s.tar.bz2' % \
-
(_build.main_package().name(
-log.notice('tarball: %s' % (os.path.relpath(path.host(tar
+tarname = path.join(tardir,
+path.basename('%s.tar.bz2' % (self.bset)))
+log.notice('tarfile: %s' % (os.path.relpath(path.host(tarname
if not self.opts.dry_run():
-tmproot = _build.config.expand('%{_tmproot}')
-cmd = _build.config.expand('"cd ' + tmproot + \
- ' && %{__tar} -cf - . | %{__bzip2}
> ' + \
- tar + '"')
-_build.run(cmd, shell_opts = '-c', cwd = tmproot)
+tar = None
+try:
+tar = tarfile.open(tarname, 'w:bz2')
+for filedir in sorted(path.listdir(stagingroot)):
+src = path.join(stagingroot, filedir)
+dst = path.join(prefix, filedir)
+log.trace('tar: %s -> %s' % (src, dst))
+tar.add(src, dst)
+except OSError as oe:
+raise error.general('tarfile: %s: %s' % (self.bset, oe))
+finally:
+if tar is not None:
+tar.close()
def parse(self, bset):
@@ -500,8 +513,6 @@ class buildset:
copy.copy(self.macros),
format = 'xml',
mail = mail)
-if s == len(configs) - 1
