Re: [PATCH] sb/set-builder: Fix staging and tar file generation with a single config build

2022-09-29 Thread Chris Johns
On 30/9/2022 3:33 pm, Christian MAUDERER wrote:
> Am 30.09.22 um 05:49 schrieb Chris Johns:
>> On 29/9/2022 9:50 pm, Chris Johns wrote:
>>> On 29/9/22 9:45 pm, Christian MAUDERER wrote:
 Hello Chris,

 thanks for the quick patch. With this qemu and microblaze work again like
 expected.

 I tested all tools starting with devel/* and from the ones that work only
 devel/autotools-internal didn't generate a tar archive. But that one has a
 comment "Do not use via the command line" in the bset file so that is most
 likely fine.

 Some of other devel/* packages didn't build in my test setup, but I have 
 never
 tested or used them before so that is probably a problem of my build
 environment
 or maybe a known bug.
>>>
>>> Thanks for the testing. I will push to the devel branch and 5.
>>>
>>
>> Tarfile creation is working however installing is not. I am working on fixing
>> this.
>>
>> Chris
> 
> Sorry that I missed that. I only tried to generate the tar archives.

Same. Testing a fix but it takes time to check properly.

I am wondering if I can create a test mode in the deployment repo. The hard part
is how to automatically check the build has worked.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel


Re: [PATCH] sb/set-builder: Fix staging and tar file generation with a single config build

2022-09-29 Thread Christian MAUDERER

Am 30.09.22 um 05:49 schrieb Chris Johns:

On 29/9/2022 9:50 pm, Chris Johns wrote:

On 29/9/22 9:45 pm, Christian MAUDERER wrote:

Hello Chris,

thanks for the quick patch. With this qemu and microblaze work again like 
expected.

I tested all tools starting with devel/* and from the ones that work only
devel/autotools-internal didn't generate a tar archive. But that one has a
comment "Do not use via the command line" in the bset file so that is most
likely fine.

Some of other devel/* packages didn't build in my test setup, but I have never
tested or used them before so that is probably a problem of my build environment
or maybe a known bug.


Thanks for the testing. I will push to the devel branch and 5.



Tarfile creation is working however installing is not. I am working on fixing 
this.

Chris


Sorry that I missed that. I only tried to generate the tar archives.

Best regards

Christian
--

embedded brains GmbH
Herr Christian MAUDERER
Dornierstr. 4
82178 Puchheim
Germany
email:  christian.maude...@embedded-brains.de
phone:  +49-89-18 94 741 - 18
mobile: +49-176-152 206 08

Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [PATCH] sb/set-builder: Fix staging and tar file generation with a single config build

2022-09-29 Thread Chris Johns
On 29/9/2022 9:50 pm, Chris Johns wrote:
> On 29/9/22 9:45 pm, Christian MAUDERER wrote:
>> Hello Chris,
>>
>> thanks for the quick patch. With this qemu and microblaze work again like 
>> expected.
>>
>> I tested all tools starting with devel/* and from the ones that work only
>> devel/autotools-internal didn't generate a tar archive. But that one has a
>> comment "Do not use via the command line" in the bset file so that is most
>> likely fine.
>>
>> Some of other devel/* packages didn't build in my test setup, but I have 
>> never
>> tested or used them before so that is probably a problem of my build 
>> environment
>> or maybe a known bug.
> 
> Thanks for the testing. I will push to the devel branch and 5.
> 

Tarfile creation is working however installing is not. I am working on fixing 
this.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel


Re: Git master/main/trunk/current/best...

2022-09-29 Thread Chris Johns
On 30/9/22 4:00 am, Joel Sherrill wrote:
> I'd like to propose that short term we change lwip main to master for
> consistency with existing repos. This helps avoid stupid mistakes because lwip
> is the odd case.

I propose we use devel. I prefer it over main.

When should we change away from master?

> Longer term, we may want to change the name master in all repos. But we need 
> to
> agree on a new name, socialize it, update documentation, and announce it. It's
> more than a simple set of git commands impacting only a few people.
> 
> Can we come to some agreement on what to do? 

New git versions warns you about the use of master when creating a new repo and
at a guess this is why the repo has main. I am using devel in rtems-deployment
so there will be a mix.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [PATCH] spec/pkgconfig: Allow builds to override headers

2022-09-29 Thread Chris Johns
On 29/9/22 11:24 pm, Kinsey Moore wrote:
> On 9/28/2022 19:03, Chris Johns wrote:
>> On 29/9/2022 7:13 am, Kinsey Moore wrote:
>>> This allows any builds targeting an installed RTEMS BSP to override
>>> headers in the installed BSP reliably, including headers previously
>>> installed by that or other builds. This includes applications, network
>>> stacks, libraries, and any other builds.
>> I am a little confused by these comments. This change effects the generated 
>> .pc
>> file for a BSP so it is only used once it is installed.
> Correct, this is a fix for things like rtems-libbsd and rtems-lwip that allows
> them to build correctly even if there are existing conflicting installations 
> of
> that library already installed in the BSP install.

So this is a change to aid developers of these packages who use a single prefix
for the tools, BSP and packages? I split the install paths up and that avoids
the problem.

>> An install should update
>> the headers at the same time the .pc is installed and made available so what 
>> is
>> old or previous? What are the "builds targeting" you refer too?
> "builds targeting an installed RTEMS BSP" refers to any external build that 
> uses
> installed RTEMS headers and libraries. These external builds can install their
> own files in the BSP install.

Is this install or reinstall?

>> I think defining the include search of RTEMS BSP and any vertical stack 
>> packages
>> headers installed under the same prefix as system headers seems like the 
>> right
>> thing to do. However this change will silence warnings from RTEMS (and 
>> installed
>> packages). Is that want we want?
> 
> What warnings will this silence? 

https://gcc.gnu.org/onlinedocs/cpp/System-Headers.html

This changes the level of warnings we currently have for a specific but
important group of packages that are based on rtems_waf or use .pc files. I
think this is important to consider.

> It shouldn't affect RTEMS builds because RTEMS
> doesn't use the pkgconfg while building. It still places installed headers
> before actual system/tools headers in the include hierarchy, so any build 
> errors
> generated that way should be preserved.

Is Makefile.inc also updated? It effects some users of RTEMS but not all. Is
that important?

Is this something we should document as mandated for all users of BSP installed
headers?

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel


Git master/main/trunk/current/best...

2022-09-29 Thread Joel Sherrill
Hi

I've been bitten twice no because lwip uses main instead of master.
Although I am not offended by having the trunk be named master, I
understand others do care.

I'd like to propose that short term we change lwip main to master for
consistency with existing repos. This helps avoid stupid mistakes because
lwip is the odd case.

Longer term, we may want to change the name master in all repos. But we
need to agree on a new name, socialize it, update documentation, and
announce it. It's more than a simple set of git commands impacting only a
few people.

Can we come to some agreement on what to do?

Thanks

--joel
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

RE: [PATCH] bsps: Improve riscv console FDT parsing

2022-09-29 Thread Alan Cudmore
Hi Padmarao,Could you try this patch on your Polarfire board? It works on the generic QEMU BSP and the BSP I am working on which uses the FRDME310ARTY/SiFive UART. It builds with the Polarfire BSP, but I am not able to test it. I downloaded and built QEMU that has Polarfire support, but I need to download and install SoftConsole to get the rest of the parts I need to prepare the binary.Thanks!Alan  From: Alan CudmoreSent: Thursday, September 29, 2022 12:12 PMTo: devel@rtems.orgCc: Alan CudmoreSubject: [PATCH] bsps: Improve riscv console FDT parsing This fixes a problem with parsing the FDT compatible property byreplacing the RISCV_CONSOLE_IS_COMPATIBLE macro with calls tothe fdt_stringlist_contains function. The macro only works whenthe compatible FDT entry is a single string and not a list ofstrings. The new call will compare each item in the string list. Close #4728.--- bsps/riscv/riscv/console/console-config.c | 14 +- 1 file changed, 5 insertions(+), 9 deletions(-) diff --git a/bsps/riscv/riscv/console/console-config.c b/bsps/riscv/riscv/console/console-config.cindex d962a5a418..7908c2f325 100644--- a/bsps/riscv/riscv/console/console-config.c+++ b/bsps/riscv/riscv/console/console-config.c@@ -139,10 +139,6 @@ static void riscv_console_set_reg_32(uintptr_t addr, uint8_t i, uint8_t val) } #endif -#define RISCV_CONSOLE_IS_COMPATIBLE(actual, actual_len, desired) \-  (actual_len == sizeof(desired) \- && memcmp(actual, desired, sizeof(desired) - 1) == 0)- static void riscv_console_probe(void) {   const void *fdt;@@ -170,7 +166,7 @@ static void riscv_console_probe(void) }  #if RISCV_ENABLE_HTIF_SUPPORT != 0-    if (RISCV_CONSOLE_IS_COMPATIBLE(compat, compat_len, "ucb,htif0")) {+    if (fdt_stringlist_contains(compat, compat_len, "ucb,htif0")) {   htif_console_context_init(_console_instance.base, node);    riscv_console.context = _console_instance.base;@@ -181,8 +177,8 @@ static void riscv_console_probe(void)  #if RISCV_CONSOLE_MAX_NS16550_DEVICES > 0 if (-  (RISCV_CONSOLE_IS_COMPATIBLE(compat, compat_len, "ns16550a")-  || RISCV_CONSOLE_IS_COMPATIBLE(compat, compat_len, "ns16750"))+    (fdt_stringlist_contains(compat, compat_len, "ns16550a")+    || fdt_stringlist_contains(compat, compat_len, "ns16750")) && ns16550_devices < RISCV_CONSOLE_MAX_NS16550_DEVICES ) {   ns16550_context *ctx;@@ -203,7 +199,7 @@ static void riscv_console_probe(void) ctx->set_reg = riscv_console_set_reg_8;   } -  if (RISCV_CONSOLE_IS_COMPATIBLE(compat, compat_len, "ns16750")) {+  if (fdt_stringlist_contains(compat, compat_len, "ns16750")) { ctx->has_precision_clock_synthesizer = true;   } @@ -243,7 +239,7 @@ static void riscv_console_probe(void) #endif  #if RISCV_ENABLE_FRDME310ARTY_SUPPORT != 0-    if (RISCV_CONSOLE_IS_COMPATIBLE(compat, compat_len, "sifive,uart0")) {+    if (fdt_stringlist_contains(compat, compat_len, "sifive,uart0")) {   fe310_uart_context *ctx;    ctx = _uart_instance;-- 2.34.1  
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

[PATCH] bsps: Improve riscv console FDT parsing

2022-09-29 Thread Alan Cudmore
This fixes a problem with parsing the FDT compatible property by
replacing the RISCV_CONSOLE_IS_COMPATIBLE macro with calls to
the fdt_stringlist_contains function. The macro only works when
the compatible FDT entry is a single string and not a list of
strings. The new call will compare each item in the string list.

Close #4728.
---
 bsps/riscv/riscv/console/console-config.c | 14 +-
 1 file changed, 5 insertions(+), 9 deletions(-)

diff --git a/bsps/riscv/riscv/console/console-config.c 
b/bsps/riscv/riscv/console/console-config.c
index d962a5a418..7908c2f325 100644
--- a/bsps/riscv/riscv/console/console-config.c
+++ b/bsps/riscv/riscv/console/console-config.c
@@ -139,10 +139,6 @@ static void riscv_console_set_reg_32(uintptr_t addr, 
uint8_t i, uint8_t val)
 }
 #endif
 
-#define RISCV_CONSOLE_IS_COMPATIBLE(actual, actual_len, desired) \
-  (actual_len == sizeof(desired) \
- && memcmp(actual, desired, sizeof(desired) - 1) == 0)
-
 static void riscv_console_probe(void)
 {
   const void *fdt;
@@ -170,7 +166,7 @@ static void riscv_console_probe(void)
 }
 
 #if RISCV_ENABLE_HTIF_SUPPORT != 0
-if (RISCV_CONSOLE_IS_COMPATIBLE(compat, compat_len, "ucb,htif0")) {
+if (fdt_stringlist_contains(compat, compat_len, "ucb,htif0")) {
   htif_console_context_init(_console_instance.base, node);
 
   riscv_console.context = _console_instance.base;
@@ -181,8 +177,8 @@ static void riscv_console_probe(void)
 
 #if RISCV_CONSOLE_MAX_NS16550_DEVICES > 0
 if (
-  (RISCV_CONSOLE_IS_COMPATIBLE(compat, compat_len, "ns16550a")
-  || RISCV_CONSOLE_IS_COMPATIBLE(compat, compat_len, "ns16750"))
+(fdt_stringlist_contains(compat, compat_len, "ns16550a")
+|| fdt_stringlist_contains(compat, compat_len, "ns16750"))
 && ns16550_devices < RISCV_CONSOLE_MAX_NS16550_DEVICES
 ) {
   ns16550_context *ctx;
@@ -203,7 +199,7 @@ static void riscv_console_probe(void)
 ctx->set_reg = riscv_console_set_reg_8;
   }
 
-  if (RISCV_CONSOLE_IS_COMPATIBLE(compat, compat_len, "ns16750")) {
+  if (fdt_stringlist_contains(compat, compat_len, "ns16750")) {
 ctx->has_precision_clock_synthesizer = true;
   }
 
@@ -243,7 +239,7 @@ static void riscv_console_probe(void)
 #endif
 
 #if RISCV_ENABLE_FRDME310ARTY_SUPPORT != 0
-if (RISCV_CONSOLE_IS_COMPATIBLE(compat, compat_len, "sifive,uart0")) {
+if (fdt_stringlist_contains(compat, compat_len, "sifive,uart0")) {
   fe310_uart_context *ctx;
 
   ctx = _uart_instance;
-- 
2.34.1

___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel


Re: [PATCH] spec/pkgconfig: Allow builds to override headers

2022-09-29 Thread Kinsey Moore

On 9/28/2022 19:03, Chris Johns wrote:

On 29/9/2022 7:13 am, Kinsey Moore wrote:

This allows any builds targeting an installed RTEMS BSP to override
headers in the installed BSP reliably, including headers previously
installed by that or other builds. This includes applications, network
stacks, libraries, and any other builds.

I am a little confused by these comments. This change effects the generated .pc
file for a BSP so it is only used once it is installed.
Correct, this is a fix for things like rtems-libbsd and rtems-lwip that 
allows them to build correctly even if there are existing conflicting 
installations of that library already installed in the BSP install.

An install should update
the headers at the same time the .pc is installed and made available so what is
old or previous? What are the "builds targeting" you refer too?
"builds targeting an installed RTEMS BSP" refers to any external build 
that uses installed RTEMS headers and libraries. These external builds 
can install their own files in the BSP install.


I think defining the include search of RTEMS BSP and any vertical stack packages
headers installed under the same prefix as system headers seems like the right
thing to do. However this change will silence warnings from RTEMS (and installed
packages). Is that want we want?


What warnings will this silence? It shouldn't affect RTEMS builds 
because RTEMS doesn't use the pkgconfg while building. It still places 
installed headers before actual system/tools headers in the include 
hierarchy, so any build errors generated that way should be preserved.



Kinsey

___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel


Re: [PATCH] sb/execute: Use a decoder that maintains state aross blocks

2022-09-29 Thread Frank Kühndel

Hello Chris,

I reviewed this patch and I also tested it in the environment which made 
problems earlier. The UnicodeDecodeError disappeared. This patch is fine 
for me.


Many tanks for fix this issue.
Frank

On 9/29/22 12:59, chr...@rtems.org wrote:

From: Chris Johns

Update #4726
---
  source-builder/sb/execute.py | 4 +++-
  1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/source-builder/sb/execute.py b/source-builder/sb/execute.py
index 3db9abc..bdab373 100755
--- a/source-builder/sb/execute.py
+++ b/source-builder/sb/execute.py
@@ -27,6 +27,7 @@
  from __future__ import print_function
  
  import functools

+import codecs
  import io
  import os
  import re
@@ -181,6 +182,7 @@ class execute(object):
  
  if trace_threads:

  print('execute:_readthread: start')
+decoder = codecs.getincrementaldecoder(sys.stdout.encoding)()
  count = 0
  line = ''
  try:
@@ -201,7 +203,7 @@ class execute(object):
  break
  # str and bytes are the same type in Python2
  if type(data) is not str and type(data) is bytes:
-data = data.decode(sys.stdout.encoding)
+data = decoder.decode(data)
  last_ch = data[-1]
  sd = (line + data).split('\n')
  if last_ch != '\n':


--
embedded brains GmbH
Herr Frank KÜHNDEL
Dornierstr. 4
82178 Puchheim
Germany
email: frank.kuehn...@embedded-brains.de
phone:  +49-89-18 94 741 - 23
mobile: +49-176-15 22 06 - 11
fax:+49-89-18 94 741 - 08

Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/

___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [PATCH] sb/set-builder: Fix staging and tar file generation with a single config build

2022-09-29 Thread Chris Johns
On 29/9/22 9:45 pm, Christian MAUDERER wrote:
> Hello Chris,
> 
> thanks for the quick patch. With this qemu and microblaze work again like 
> expected.
> 
> I tested all tools starting with devel/* and from the ones that work only
> devel/autotools-internal didn't generate a tar archive. But that one has a
> comment "Do not use via the command line" in the bset file so that is most
> likely fine.
> 
> Some of other devel/* packages didn't build in my test setup, but I have never
> tested or used them before so that is probably a problem of my build 
> environment
> or maybe a known bug.

Thanks for the testing. I will push to the devel branch and 5.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel


Re: [PATCH] sb/set-builder: Fix staging and tar file generation with a single config build

2022-09-29 Thread Christian MAUDERER

Hello Chris,

thanks for the quick patch. With this qemu and microblaze work again 
like expected.


I tested all tools starting with devel/* and from the ones that work 
only devel/autotools-internal didn't generate a tar archive. But that 
one has a comment "Do not use via the command line" in the bset file so 
that is most likely fine.


Some of other devel/* packages didn't build in my test setup, but I have 
never tested or used them before so that is probably a problem of my 
build environment or maybe a known bug.


Best regards

Christian

Am 29.09.22 um 10:52 schrieb chr...@rtems.org:

From: Chris Johns 

Closes #4730
---
  source-builder/sb/setbuilder.py | 22 ++
  1 file changed, 6 insertions(+), 16 deletions(-)

diff --git a/source-builder/sb/setbuilder.py b/source-builder/sb/setbuilder.py
index 3e16111..5921eed 100644
--- a/source-builder/sb/setbuilder.py
+++ b/source-builder/sb/setbuilder.py
@@ -433,19 +433,11 @@ class buildset:
  interrupted = False
  
  #

-# If this is the outter most buildset it's files are installed. Nested
-# build sets staged their installed file. The staged files are install
-# when the outtter most build finishes.
+# If installing switch to staging. Not sure if this is still
+# needed.
  #
-if nesting_count != 1:
-if self.installing():
-self.macros['install_mode'] = 'staging'
-
-#
-# Only the outter build set can have staging to install. Get the 
staging
-# root via the config because it could require a valid config.
-#
-have_staging = False
+if self.installing():
+self.macros['install_mode'] = 'staging'
  
  try:

  configs = self.load()
@@ -453,7 +445,7 @@ class buildset:
  log.trace('_bset: %2d: %s: configs: %s'  % (nesting_count,
  self.bset, ', 
'.join(configs)))
  
-if nesting_count == 1 and len(configs) > 1:

+if nesting_count == 1:
  #
  # Prepend staging areas, bin directory to the
  # path. Lets the later package depend on the earlier
@@ -485,8 +477,6 @@ class buildset:
   '=' * (74 - 
len(configs[s]
  bs = buildset(configs[s], self.configs, opts, macros)
  bs.build(deps, nesting_count, mail)
-if self.installing():
-have_staging = True
  del bs
  elif configs[s].endswith('.cfg'):
  if mail:
@@ -620,7 +610,7 @@ class buildset:
  #
  # If builds have been staged install into the final prefix.
  #
-if have_staging and not have_errors:
+if not have_errors:
  stagingroot = macro_expand(self.macros, '%{stagingroot}')
  have_stagingroot = path.exists(stagingroot)
  do_install = not self.opts.no_install()


--

embedded brains GmbH
Herr Christian MAUDERER
Dornierstr. 4
82178 Puchheim
Germany
email:  christian.maude...@embedded-brains.de
phone:  +49-89-18 94 741 - 18
mobile: +49-176-152 206 08

Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

[PATCH] sb/execute: Use a decoder that maintains state aross blocks

2022-09-29 Thread chrisj
From: Chris Johns 

Update #4726
---
 source-builder/sb/execute.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/source-builder/sb/execute.py b/source-builder/sb/execute.py
index 3db9abc..bdab373 100755
--- a/source-builder/sb/execute.py
+++ b/source-builder/sb/execute.py
@@ -27,6 +27,7 @@
 from __future__ import print_function
 
 import functools
+import codecs
 import io
 import os
 import re
@@ -181,6 +182,7 @@ class execute(object):
 
 if trace_threads:
 print('execute:_readthread: start')
+decoder = codecs.getincrementaldecoder(sys.stdout.encoding)()
 count = 0
 line = ''
 try:
@@ -201,7 +203,7 @@ class execute(object):
 break
 # str and bytes are the same type in Python2
 if type(data) is not str and type(data) is bytes:
-data = data.decode(sys.stdout.encoding)
+data = decoder.decode(data)
 last_ch = data[-1]
 sd = (line + data).split('\n')
 if last_ch != '\n':
-- 
2.37.1

___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel


[PATCH] sb/version: Set top from external package

2022-09-29 Thread chrisj
From: Chris Johns 

---
 source-builder/sb/version.py | 10 +-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/source-builder/sb/version.py b/source-builder/sb/version.py
index 29d2dc5..4ec7cfa 100644
--- a/source-builder/sb/version.py
+++ b/source-builder/sb/version.py
@@ -89,9 +89,13 @@ _version_str = '%s.%s' % (_version, _revision)
 _released = False
 _git = False
 _is_loaded = False
+_top_dir = None
 
 def _top():
-top = path.dirname(sys.argv[0])
+if _top_dir is None:
+top = path.dirname(sys.argv[0])
+else:
+top = _top_dir
 if len(top) == 0:
 top = '.'
 return top
@@ -183,6 +187,10 @@ def _load_git_version():
 _is_loaded = True
 return _git
 
+def set_top(top):
+global _top_dir
+_top_dir = top
+
 def load_release_settings(section, error = False):
 vc, v = _load_released_version_config()
 items = []
-- 
2.37.1

___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel


[PATCH] sb/set-builder: Fix staging and tar file generation with a single config build

2022-09-29 Thread chrisj
From: Chris Johns 

Closes #4730
---
 source-builder/sb/setbuilder.py | 22 ++
 1 file changed, 6 insertions(+), 16 deletions(-)

diff --git a/source-builder/sb/setbuilder.py b/source-builder/sb/setbuilder.py
index 3e16111..5921eed 100644
--- a/source-builder/sb/setbuilder.py
+++ b/source-builder/sb/setbuilder.py
@@ -433,19 +433,11 @@ class buildset:
 interrupted = False
 
 #
-# If this is the outter most buildset it's files are installed. Nested
-# build sets staged their installed file. The staged files are install
-# when the outtter most build finishes.
+# If installing switch to staging. Not sure if this is still
+# needed.
 #
-if nesting_count != 1:
-if self.installing():
-self.macros['install_mode'] = 'staging'
-
-#
-# Only the outter build set can have staging to install. Get the 
staging
-# root via the config because it could require a valid config.
-#
-have_staging = False
+if self.installing():
+self.macros['install_mode'] = 'staging'
 
 try:
 configs = self.load()
@@ -453,7 +445,7 @@ class buildset:
 log.trace('_bset: %2d: %s: configs: %s'  % (nesting_count,
 self.bset, ', 
'.join(configs)))
 
-if nesting_count == 1 and len(configs) > 1:
+if nesting_count == 1:
 #
 # Prepend staging areas, bin directory to the
 # path. Lets the later package depend on the earlier
@@ -485,8 +477,6 @@ class buildset:
  '=' * (74 - 
len(configs[s]
 bs = buildset(configs[s], self.configs, opts, macros)
 bs.build(deps, nesting_count, mail)
-if self.installing():
-have_staging = True
 del bs
 elif configs[s].endswith('.cfg'):
 if mail:
@@ -620,7 +610,7 @@ class buildset:
 #
 # If builds have been staged install into the final prefix.
 #
-if have_staging and not have_errors:
+if not have_errors:
 stagingroot = macro_expand(self.macros, '%{stagingroot}')
 have_stagingroot = path.exists(stagingroot)
 do_install = not self.opts.no_install()
-- 
2.37.1

___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel


Re: [rtems-source-builder commit] sb/setbuilder: Correctly create build set tar files

2022-09-29 Thread Chris Johns
On 29/9/22 6:01 pm, Christian MAUDERER wrote:
> Am 29.09.22 um 09:52 schrieb Chris Johns:
>>
>>
>> On 29/9/22 5:13 pm, Christian MAUDERER wrote:
>>> Am 29.09.22 um 08:56 schrieb Chris Johns:
 On 29/9/2022 4:55 pm, Christian MAUDERER wrote:
> Am 29.09.22 um 08:54 schrieb Chris Johns:
>> On 29/9/2022 4:42 pm, Christian MAUDERER wrote:
 It could be a bug if the tools builds work, ie 6/rtems-*. Please raise 
 a
 ticket?
>>>
>>> The tool builds work except for the 6/rtems-microblaze.
>>
>> Thanks, I will take a look.
>>
>
> I just checked it: There is the same behavior for devel/dtc (which is much
> faster to build). Without the patch an archive is created. With the patch 
> it
> doesn't create one.

 Awesome, that will help.

>>>
>>> In case of dtc, it seems to not work because dtc is in
>>>
>>>    devel/dtc-1.6.1-1.cfg
>>>
>>> which ends in a ".cfg" and not in a ".bset".
>>>
>>> The bset_tar(...) function is only called if the have_staging is set here:
>>>
>>> https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/setbuilder.py?id=161b7f108c3daa0f71ac289ba048a68b730d422c#n623
>>>
>>>
>>>
>>> That variable seems to be only set to True here:
>>>
>>> https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/setbuilder.py?id=161b7f108c3daa0f71ac289ba048a68b730d422c#n489
>>>
>>>
>>>
>>> And that statement is in a
>>>
>>>    if configs[s].endswith('.bset')
>>>
>>> Does that mean that only bsets can be packed?
>>
>> It means the build is not being staged and so no bset tar generation. At a 
>> guess
>> a single package in a build does not need to be staged so that step is 
>> missed.
>>
>> Just a bug. If you could please raise a ticket and assign to me I will take 
>> care
>> of this.
>>
> 
> https://devel.rtems.org/ticket/4730#ticket

Thanks.

> Is there anything I can help with solving that?

I have a fix so some testing a bit would be a big help.

The staging logic was a progression of changes at the time and took some effort.
Once it was sorted some extra and unneeded complexity was left and that can be
remove. All bsets and cfgs should be staged so removing the logic that made only
nested builds stage fixes it for dtc. I just need to test the more complex 
builds.

The microblaze is a separate gcc version so I am not sure about it. I will to
look into it next.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [rtems-source-builder commit] sb/setbuilder: Correctly create build set tar files

2022-09-29 Thread Christian MAUDERER

Am 29.09.22 um 09:52 schrieb Chris Johns:



On 29/9/22 5:13 pm, Christian MAUDERER wrote:

Am 29.09.22 um 08:56 schrieb Chris Johns:

On 29/9/2022 4:55 pm, Christian MAUDERER wrote:

Am 29.09.22 um 08:54 schrieb Chris Johns:

On 29/9/2022 4:42 pm, Christian MAUDERER wrote:

It could be a bug if the tools builds work, ie 6/rtems-*. Please raise a
ticket?


The tool builds work except for the 6/rtems-microblaze.


Thanks, I will take a look.



I just checked it: There is the same behavior for devel/dtc (which is much
faster to build). Without the patch an archive is created. With the patch it
doesn't create one.


Awesome, that will help.



In case of dtc, it seems to not work because dtc is in

   devel/dtc-1.6.1-1.cfg

which ends in a ".cfg" and not in a ".bset".

The bset_tar(...) function is only called if the have_staging is set here:

https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/setbuilder.py?id=161b7f108c3daa0f71ac289ba048a68b730d422c#n623


That variable seems to be only set to True here:

https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/setbuilder.py?id=161b7f108c3daa0f71ac289ba048a68b730d422c#n489


And that statement is in a

   if configs[s].endswith('.bset')

Does that mean that only bsets can be packed?


It means the build is not being staged and so no bset tar generation. At a guess
a single package in a build does not need to be staged so that step is missed.

Just a bug. If you could please raise a ticket and assign to me I will take care
of this.



https://devel.rtems.org/ticket/4730#ticket

Is there anything I can help with solving that?

Best regards

Christian


Nice work getting it down to here. I appreciate it. >

I think having a tarball for tools like qemu or dtc could be quite useful too.


Agreed.


PS: Not sure yet why microblaze is affected too. That maybe is another case.


Thanks. Getting to it with rtems-deployment repo but I hit 5 verses devel branch
issues.

Chris


--

embedded brains GmbH
Herr Christian MAUDERER
Dornierstr. 4
82178 Puchheim
Germany
email:  christian.maude...@embedded-brains.de
phone:  +49-89-18 94 741 - 18
mobile: +49-176-152 206 08

Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [rtems-source-builder commit] sb/setbuilder: Correctly create build set tar files

2022-09-29 Thread Chris Johns


On 29/9/22 5:13 pm, Christian MAUDERER wrote:
> Am 29.09.22 um 08:56 schrieb Chris Johns:
>> On 29/9/2022 4:55 pm, Christian MAUDERER wrote:
>>> Am 29.09.22 um 08:54 schrieb Chris Johns:
 On 29/9/2022 4:42 pm, Christian MAUDERER wrote:
>> It could be a bug if the tools builds work, ie 6/rtems-*. Please raise a
>> ticket?
>
> The tool builds work except for the 6/rtems-microblaze.

 Thanks, I will take a look.

>>>
>>> I just checked it: There is the same behavior for devel/dtc (which is much
>>> faster to build). Without the patch an archive is created. With the patch it
>>> doesn't create one.
>>
>> Awesome, that will help.
>>
> 
> In case of dtc, it seems to not work because dtc is in
> 
>   devel/dtc-1.6.1-1.cfg
> 
> which ends in a ".cfg" and not in a ".bset".
> 
> The bset_tar(...) function is only called if the have_staging is set here:
> 
> https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/setbuilder.py?id=161b7f108c3daa0f71ac289ba048a68b730d422c#n623
> 
> 
> That variable seems to be only set to True here:
> 
> https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/setbuilder.py?id=161b7f108c3daa0f71ac289ba048a68b730d422c#n489
> 
> 
> And that statement is in a
> 
>   if configs[s].endswith('.bset')
> 
> Does that mean that only bsets can be packed? 

It means the build is not being staged and so no bset tar generation. At a guess
a single package in a build does not need to be staged so that step is missed.

Just a bug. If you could please raise a ticket and assign to me I will take care
of this.

Nice work getting it down to here. I appreciate it.

> I think having a tarball for tools like qemu or dtc could be quite useful too.

Agreed.

> PS: Not sure yet why microblaze is affected too. That maybe is another case.

Thanks. Getting to it with rtems-deployment repo but I hit 5 verses devel branch
issues.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [rtems-source-builder commit] sb/setbuilder: Correctly create build set tar files

2022-09-29 Thread Christian MAUDERER

Am 29.09.22 um 08:56 schrieb Chris Johns:

On 29/9/2022 4:55 pm, Christian MAUDERER wrote:

Am 29.09.22 um 08:54 schrieb Chris Johns:

On 29/9/2022 4:42 pm, Christian MAUDERER wrote:

It could be a bug if the tools builds work, ie 6/rtems-*. Please raise a
ticket?


The tool builds work except for the 6/rtems-microblaze.


Thanks, I will take a look.



I just checked it: There is the same behavior for devel/dtc (which is much
faster to build). Without the patch an archive is created. With the patch it
doesn't create one.


Awesome, that will help.



In case of dtc, it seems to not work because dtc is in

  devel/dtc-1.6.1-1.cfg

which ends in a ".cfg" and not in a ".bset".

The bset_tar(...) function is only called if the have_staging is set here:

https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/setbuilder.py?id=161b7f108c3daa0f71ac289ba048a68b730d422c#n623

That variable seems to be only set to True here:

https://git.rtems.org/rtems-source-builder/tree/source-builder/sb/setbuilder.py?id=161b7f108c3daa0f71ac289ba048a68b730d422c#n489

And that statement is in a

  if configs[s].endswith('.bset')

Does that mean that only bsets can be packed? I think having a tarball 
for tools like qemu or dtc could be quite useful too.


PS: Not sure yet why microblaze is affected too. That maybe is another case.

Best regards

Christian
--

embedded brains GmbH
Herr Christian MAUDERER
Dornierstr. 4
82178 Puchheim
Germany
email:  christian.maude...@embedded-brains.de
phone:  +49-89-18 94 741 - 18
mobile: +49-176-152 206 08

Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [rtems-source-builder commit] sb/setbuilder: Correctly create build set tar files

2022-09-29 Thread Chris Johns
On 29/9/2022 4:55 pm, Christian MAUDERER wrote:
> Am 29.09.22 um 08:54 schrieb Chris Johns:
>> On 29/9/2022 4:42 pm, Christian MAUDERER wrote:
 It could be a bug if the tools builds work, ie 6/rtems-*. Please raise a
 ticket?
>>>
>>> The tool builds work except for the 6/rtems-microblaze.
>>
>> Thanks, I will take a look.
>>
> 
> I just checked it: There is the same behavior for devel/dtc (which is much
> faster to build). Without the patch an archive is created. With the patch it
> doesn't create one.

Awesome, that will help.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel


Re: [rtems-source-builder commit] sb/setbuilder: Correctly create build set tar files

2022-09-29 Thread Christian MAUDERER

Am 29.09.22 um 08:54 schrieb Chris Johns:

On 29/9/2022 4:42 pm, Christian MAUDERER wrote:

It could be a bug if the tools builds work, ie 6/rtems-*. Please raise a ticket?


The tool builds work except for the 6/rtems-microblaze.


Thanks, I will take a look.



I just checked it: There is the same behavior for devel/dtc (which is 
much faster to build). Without the patch an archive is created. With the 
patch it doesn't create one.


Best regards

Christian

--

embedded brains GmbH
Herr Christian MAUDERER
Dornierstr. 4
82178 Puchheim
Germany
email:  christian.maude...@embedded-brains.de
phone:  +49-89-18 94 741 - 18
mobile: +49-176-152 206 08

Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel

Re: [rtems-source-builder commit] sb/setbuilder: Correctly create build set tar files

2022-09-29 Thread Chris Johns
On 29/9/2022 4:42 pm, Christian MAUDERER wrote:
>> It could be a bug if the tools builds work, ie 6/rtems-*. Please raise a 
>> ticket?
> 
> The tool builds work except for the 6/rtems-microblaze.

Thanks, I will take a look.

Chris
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel


Re: [rtems-source-builder commit] sb/setbuilder: Correctly create build set tar files

2022-09-29 Thread Christian MAUDERER

Hello Chris,

thanks for the response.

Am 29.09.22 um 01:40 schrieb Chris Johns:

On 28/9/2022 11:42 pm, Christian MAUDERER wrote:

Hello,

with this patch, I don't get a tar for devel/qemu and for the 6/rtems-microblaze
anymore. All other 6/rtems-* toolchains work without problems. I haven't tested
a lot of the other packages.

The bset for microblaze is a bit different from the other ones. But I'm not yet
sure what the relevant difference is.

@Chris: With your change: What is necessary that a bset can generate a tar 
archive?


It could be a bug if the tools builds work, ie 6/rtems-*. Please raise a ticket?


The tool builds work except for the 6/rtems-microblaze.



I changed the --bset-tar-file option to create a single tarfile of the final
staged output of a build. There were a few commits to get this right so I assume
you are testing on the latest?


I used the current master during the tests. I reverted only the single 
"Correctly create build set tar files" commit to check whether it was 
the relevant one. Without this single commit but all others still in 
place the qemu tar file is created.




Incremental tarfiles based on separate buildsets in a buildset may have worked
in some cases however there were some basic issues in how it was implemented.
When you add deployment requirements on top it did not match up well and it was
confusing. The best solution was to rebase the tarfile against the final staged
output as that is known to be correct in all cases.


I'm not sure whether I understand that, but that is because I never 
analyzed how the source builder generates the tar files. I'll try to 
read a bit more documentation and sources.


Best regards

Christian



I have created a https://git.rtems.org/chrisj/rtems-deployment.git repo and in
it a config test directory
(https://git.rtems.org/chrisj/rtems-deployment.git/tree/config/test). I will
look at adding a test to make sure we catch any issues.

Thanks
Chris


--

embedded brains GmbH
Herr Christian MAUDERER
Dornierstr. 4
82178 Puchheim
Germany
email:  christian.maude...@embedded-brains.de
phone:  +49-89-18 94 741 - 18
mobile: +49-176-152 206 08

Registergericht: Amtsgericht München
Registernummer: HRB 157899
Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler
Unsere Datenschutzerklärung finden Sie hier:
https://embedded-brains.de/datenschutzerklaerung/
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel