Westernization of RTEMS/stm32H7 [was: Re: Discussion: How to handle HALs, SDKs and libraries]
Hello, Christian mentioned Zephyr west tool as one of good/possible candidates for RTEMS tree modularization especially with regarding to HALs and other 3rd party libraries. For comparison with my previous git submodulization of RTEMS/stm32H7 I've also westernized the same in a similar way. Yes, it is intended to remove CMSIS v4 *just* to proof compilation is working with CMSIS 5 from outside module! This is just dirty PoC nothing more. A tree structure looks a bit different as (folowing zephyr example) I keep modules outside the rtems tree. Hence result for west init command below is tree: modules/hal/stm32h7/stm32h7xx_hal_driver modules/hal/stm32h7/cmsis_device_h7 modules/hal/stm32h7/CMSIS_5 rtems/... So if you like to see thing, then install west: $ pip3 install --user west $ export PATH=$PATH:$HOME/.local/bin go into some testing directory and: $ west init -m https://github.com/karelfv/rtems --mr rtems-west-stm32h7-hal rtems-workspace $ cd rtems-workspace/ $ west update Compilation again produces working samples. In comparison with git submodule approach I find west a bit more civilized (user friendly). Nothing is shooting in my own feet etc. What may resonate in RTEMS devs is: - yaml format for module configuration - west implemented in python3 - west extensible It probably will not be big issue to implement custom: - west bsplist - west bspdefaults - west configure - west build which would just delegate to waf for actual work if needed... Comments welcome! Karel ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: Discussion: How to handle HALs, SDKs and libraries
On 7/15/23 16:00, o...@c-mauderer.de wrote: We should avoid rebasing and overwriting the old branch because we have to preserve a version that can be used with older RTEMS versions. So for rebasing, we would have to use multiple branches. Instead of rtems-6-branch, it could be a rtems- or something similar. Thanks for confirmation. Have the same feeling about it. I think that depends heavily on how the upstream project works. If the upstream project uses a typical git workflow and is quite active, merging is most likely the better solution because we don't get big amounts of branches with that. But this means we still merge to some rtems specific branch since master is used to sync with upstream, right? On the other hand I know that some semiconductor manufacturers tend to just dump code into a repo and they don't care whether files move around or change heavily. They happily mix changes with reformatting the complete code. In that case you have no chance of a useful merge-commit. I think in these cases it's most likely better to create a new branch on every update and rebase the patches. Anyway, still this is a branch based workflow. How would you do that workflow with just fixed commit IDs? I'm asking since so far commit IDs on submodules were always obtain from master branch which is what we basically fork and does not give us ability to patch with RTEMS changes... So I'm curious how would you do that? I think I have put that in the wrong words. I assumed you mean that you want to use some feature that always uses the latest commit of a branch (also I don't know whether git even has that feature). What I wanted to say is that if I check out a RTEMS version from two months back, it should check out the revisions of the submodules that have been used in RTEMS two months back too. It must not use the latest HEAD of the branch from today. It's just part of having a version of the code that can be reproduced later. Right! Now crucial question: is that even supported in git? I'm asking since so far I've seen only: - use fixed commit ids, but those are obtained only from master branch or - use branch and then this is always branch HEAD I've not seen anything like "use fixed commit ID from branch XXX". So question for you as an git expert: are you absolutely sure that this workflow is supported by git at all? And by this I mean have you tested that actually? :-) Of course the revision from two months back has to be available in the repo. It doesn't really matter whether it's on master because the code just worked or whether it's a RTEMS specific branch because adaptions have been necessary. You "it doesn't really matter" worries me a little bit. I'm not sure this functionality is even supported hence my question above if you have tested it yourself. Thanks! Karel ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
grlib changes Fwd: New Defects reported by Coverity Scan for RTEMS
The recent grlib changes introduced a Coverity issues.
-- Forwarded message -
From:
Date: Sat, Jul 15, 2023 at 12:22 AM
Subject: New Defects reported by Coverity Scan for RTEMS
To:
Hi,
Please find the latest report on new defect(s) introduced to RTEMS found
with Coverity Scan.
1 new defect(s) introduced to RTEMS found with Coverity Scan.
New defect(s) Reported-by: Coverity Scan
Showing 1 of 1 defect(s)
** CID 1538314: Uninitialized variables (UNINIT)
/bsps/shared/grlib/uart/apbuart_cons.c: 448 in first_open()
*** CID 1538314: Uninitialized variables (UNINIT)
/bsps/shared/grlib/uart/apbuart_cons.c: 448 in first_open()
442 return false;
443 }
444
445 uart->sending = 0;
446
447 /* Turn on RX interrupts */
>>> CID 1538314: Uninitialized variables (UNINIT)
>>> Using uninitialized value "ctrl".
448 ctrl |= APBUART_CTRL_RI;
449 if (uart->cap & CAP_DI) {
450 /* Use RX FIFO interrupt only if delayed
interrupt available. */
451 ctrl |= (APBUART_CTRL_DI | APBUART_CTRL_RF);
452 }
453 }
To view the defects in Coverity Scan visit,
https://u15810271.ct.sendgrid.net/ls/click?upn=HRESupC-2F2Czv4BOaCWWCy7my0P0qcxCbhZ31OYv50ypUUzi-2FdSNmuyRB7BEFT8xQ4-2B8hpujh0hTgQljRGId4Dg-3D-3DHAM__EU3W9teASMK00lBXX9WT4lsogDrkCcNZLvg-2FVxwAXMo-2BfWe9y-2FQVPI3NBuKOOB9pEF0PWPhWQO6FUVOcAwwdagvzNKH7AA8SymQIigXNwofEBIHyLL-2FrZJgQDgEg0O4g4HS-2FjBhvvcGAhWWb-2FxsDZrRDyXAMAUd-2BRwY7jJKszizp8zBz7IR2apEMP0z49hqJcf-2F9hLs1Dn-2B7C8NzR5v-2Bdg-3D-3D
___
build mailing list
bu...@rtems.org
http://lists.rtems.org/mailman/listinfo/build
___
devel mailing list
devel@rtems.org
http://lists.rtems.org/mailman/listinfo/devel
Re: Discussion: How to handle HALs, SDKs and libraries
Hello Karel, Am 15.07.23 um 15:03 schrieb Karel Gardas: On 7/15/23 14:33, o...@c-mauderer.de wrote: I like submodules because they are well-supported by the usual tools. Honestly. I like submodules idea, but hate its implementation. Some reasons: - submodules were added as a light weight feature and during the development of git/submodule feature they accumulated some more. - submodules started as way to get commit id of the external project to the tree. But only master branch was supported. (not suitable for RTEMS) - later submodules added capability to use tag (still not suitable for RTEMS) - later submodules added capability to use different branch from the project, but only a head of it. (finally what RTEMS need). I don't like using branches instead of fixed commit IDs. Otherwise, it's not possible to reliable re-build a fixed version. And this is most important detail, so let's discuss this alone. OK. I have a hal library, let's consider this to be https://github.com/karelfv/stm32h7xx_hal_driver -- just one project for now. This is a fork of ST Micro project of the same name which is a slow development project. Few accumulated commits over one year let say. This project needs few RTEMS related patches and there is a chance they also need to kind of adapt to the new development. So let's branch as: - master branch -- this is fork of ST Micro - rtems-6-branch -- this is a branch from master at one point of time accumulating RTEMS related changes. Now, let's assume this was done 6 months ago and now you would like to update from ST Micro upstream. How would you do that? I see: - git fetch upstream inside the master - git push master that's update of master in fork and now: - pull/merge master or - rebase master in the rtems-6-branch? I'm afraid rebase is dangerous in shared setup so probably pull/merge? We should avoid rebasing and overwriting the old branch because we have to preserve a version that can be used with older RTEMS versions. So for rebasing, we would have to use multiple branches. Instead of rtems-6-branch, it could be a rtems- or something similar. I think that depends heavily on how the upstream project works. If the upstream project uses a typical git workflow and is quite active, merging is most likely the better solution because we don't get big amounts of branches with that. On the other hand I know that some semiconductor manufacturers tend to just dump code into a repo and they don't care whether files move around or change heavily. They happily mix changes with reformatting the complete code. In that case you have no chance of a useful merge-commit. I think in these cases it's most likely better to create a new branch on every update and rebase the patches. Anyway, still this is a branch based workflow. How would you do that workflow with just fixed commit IDs? I'm asking since so far commit IDs on submodules were always obtain from master branch which is what we basically fork and does not give us ability to patch with RTEMS changes... So I'm curious how would you do that? I think I have put that in the wrong words. I assumed you mean that you want to use some feature that always uses the latest commit of a branch (also I don't know whether git even has that feature). What I wanted to say is that if I check out a RTEMS version from two months back, it should check out the revisions of the submodules that have been used in RTEMS two months back too. It must not use the latest HEAD of the branch from today. It's just part of having a version of the code that can be reproduced later. Of course the revision from two months back has to be available in the repo. It doesn't really matter whether it's on master because the code just worked or whether it's a RTEMS specific branch because adaptions have been necessary. Best regards Christian Thanks! Karel ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: Discussion: How to handle HALs, SDKs and libraries
On 7/15/23 14:33, o...@c-mauderer.de wrote: I like submodules because they are well-supported by the usual tools. Honestly. I like submodules idea, but hate its implementation. Some reasons: - submodules were added as a light weight feature and during the development of git/submodule feature they accumulated some more. - submodules started as way to get commit id of the external project to the tree. But only master branch was supported. (not suitable for RTEMS) - later submodules added capability to use tag (still not suitable for RTEMS) - later submodules added capability to use different branch from the project, but only a head of it. (finally what RTEMS need). I don't like using branches instead of fixed commit IDs. Otherwise, it's not possible to reliable re-build a fixed version. And this is most important detail, so let's discuss this alone. I have a hal library, let's consider this to be https://github.com/karelfv/stm32h7xx_hal_driver -- just one project for now. This is a fork of ST Micro project of the same name which is a slow development project. Few accumulated commits over one year let say. This project needs few RTEMS related patches and there is a chance they also need to kind of adapt to the new development. So let's branch as: - master branch -- this is fork of ST Micro - rtems-6-branch -- this is a branch from master at one point of time accumulating RTEMS related changes. Now, let's assume this was done 6 months ago and now you would like to update from ST Micro upstream. How would you do that? I see: - git fetch upstream inside the master - git push master that's update of master in fork and now: - pull/merge master or - rebase master in the rtems-6-branch? I'm afraid rebase is dangerous in shared setup so probably pull/merge? Anyway, still this is a branch based workflow. How would you do that workflow with just fixed commit IDs? I'm asking since so far commit IDs on submodules were always obtain from master branch which is what we basically fork and does not give us ability to patch with RTEMS changes... So I'm curious how would you do that? Thanks! Karel ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: Discussion: How to handle HALs, SDKs and libraries
Hello Karel, Am 15.07.23 um 12:17 schrieb Karel Gardas: On 7/15/23 09:47, o...@c-mauderer.de wrote: Am 15.07.23 um 01:48 schrieb Karel Gardas: Hello, I've created setup where I've put updated STM32H7 HAL consisting of two submodules: - stm32h7xx_hal_driver - cmsis_device_h7 and ARM's - CMSIS_5 into hals/arm/stm32h7 to follow related BSP code location. I think I would prefer "contrib/hals", "external/hals" or some other more generic top directory name. It would allow to use that for other parts like for example zlib in "contrib/libs/zlib" in the future (if we agree on having libs like that there too). Indeed, name is not the problem here. Both looks fine, somewhat prefer external a bit. At the moment, you have absolute URLs for your submodules: url = https://github.com/ARM-software/CMSIS_5.git url = https://github.com/karelfv/stm32h7xx_hal_driver.git url = https://github.com/karelfv/cmsis_device_h7.git If we use submodules, I would go for relative paths and clone the repositories to our git.rtems.org and to all mirrors. It has the advantage that even if some upstream URL changes, we won't be affected. This is dirty PoC not a final solution. Due to this I've settled on absolute urls and not relative. For final solution relative are needed, but both url (relative/abolute) and name/location in RTEMS tree is very easy to change based on decision. I like submodules because they are well-supported by the usual tools. Honestly. I like submodules idea, but hate its implementation. Some reasons: - submodules were added as a light weight feature and during the development of git/submodule feature they accumulated some more. - submodules started as way to get commit id of the external project to the tree. But only master branch was supported. (not suitable for RTEMS) - later submodules added capability to use tag (still not suitable for RTEMS) - later submodules added capability to use different branch from the project, but only a head of it. (finally what RTEMS need). I don't like using branches instead of fixed commit IDs. Otherwise, it's not possible to reliable re-build a fixed version. - in the field you can see gits supporting different sets of submodule features described above. Yes, it will force us to use a minimum git version that supports the necessary features. From my point of view, it's mainly something we should document. - submodules are easy to be broken. Personal experience from Haskell compiler (GHC) project which uses submodules a lot. I have git 2.25 and I always have an issue to pull from GHC repo since alwyas for me one submodule (when changed) will be broken for me. Conclusion: clone with submodules may kind of work, but later pull may be tricky. Agreed: It's easy to miss that a submodule is out of date. - besides submodules are not supported in 'got' which is 3rd party independent implementation of git like SCM with more sane UI I would say that this is a problem of the third-party client. We should make sure to support the official git client. Maybe additionally some (python?) git library if it is necessary for our tools. - due to submodules trickiness development policy for external integration with project code modification (aka HAL patches) needs to be very well written. I smell horrible increase in cognitive load here... Nonetheless, we should take a look at some of the reasons why other projects decided against them. For example Google has the "repo" tool used in Android. Zephyr has the "west" tool. OpenWRT uses "feeds" for something similar. I'm sure there are more. https://gerrit.googlesource.com/git-repo/ https://docs.zephyrproject.org/latest/develop/west/why.html# https://openwrt.org/docs/guide-developer/feeds All solve a similar problem. Part of them even overlap already with functionality from our waf based build system. We should consider whether we need any of the functionality of these tools and whether we want to re-implement it or whether we just want to use one of the tools and integrate with it. Oh, do I sense starting academic debate now? The point of my dirty PoC was to ground that a bit and show real stuff to get a feel of possibilities. I think the Proof of Concept is great and important. Thanks for doing it. I'm fully OK with submodules as a general direction. I just wanted to point out that we should take at least a short look at other systems that solve similar problems. If they work around serious limitations of submodules we can learn from them and avoid that we hit the same limitations. I used submodules myself in a lot of cases. And there are mainly two problems that I have: 1. I often miss pushing a submodule when I push the base repo. And I only note it way later if I try to update the repo on another PC. That happened to me multiple times when working with submodules. But another tool will just have other problems so
Re: Discussion: How to handle HALs, SDKs and libraries
On 7/15/23 09:47, o...@c-mauderer.de wrote: Am 15.07.23 um 01:48 schrieb Karel Gardas: Hello, I've created setup where I've put updated STM32H7 HAL consisting of two submodules: - stm32h7xx_hal_driver - cmsis_device_h7 and ARM's - CMSIS_5 into hals/arm/stm32h7 to follow related BSP code location. I think I would prefer "contrib/hals", "external/hals" or some other more generic top directory name. It would allow to use that for other parts like for example zlib in "contrib/libs/zlib" in the future (if we agree on having libs like that there too). Indeed, name is not the problem here. Both looks fine, somewhat prefer external a bit. At the moment, you have absolute URLs for your submodules: url = https://github.com/ARM-software/CMSIS_5.git url = https://github.com/karelfv/stm32h7xx_hal_driver.git url = https://github.com/karelfv/cmsis_device_h7.git If we use submodules, I would go for relative paths and clone the repositories to our git.rtems.org and to all mirrors. It has the advantage that even if some upstream URL changes, we won't be affected. This is dirty PoC not a final solution. Due to this I've settled on absolute urls and not relative. For final solution relative are needed, but both url (relative/abolute) and name/location in RTEMS tree is very easy to change based on decision. I like submodules because they are well-supported by the usual tools. Honestly. I like submodules idea, but hate its implementation. Some reasons: - submodules were added as a light weight feature and during the development of git/submodule feature they accumulated some more. - submodules started as way to get commit id of the external project to the tree. But only master branch was supported. (not suitable for RTEMS) - later submodules added capability to use tag (still not suitable for RTEMS) - later submodules added capability to use different branch from the project, but only a head of it. (finally what RTEMS need). - in the field you can see gits supporting different sets of submodule features described above. - submodules are easy to be broken. Personal experience from Haskell compiler (GHC) project which uses submodules a lot. I have git 2.25 and I always have an issue to pull from GHC repo since alwyas for me one submodule (when changed) will be broken for me. Conclusion: clone with submodules may kind of work, but later pull may be tricky. - besides submodules are not supported in 'got' which is 3rd party independent implementation of git like SCM with more sane UI - due to submodules trickiness development policy for external integration with project code modification (aka HAL patches) needs to be very well written. I smell horrible increase in cognitive load here... Nonetheless, we should take a look at some of the reasons why other projects decided against them. For example Google has the "repo" tool used in Android. Zephyr has the "west" tool. OpenWRT uses "feeds" for something similar. I'm sure there are more. https://gerrit.googlesource.com/git-repo/ https://docs.zephyrproject.org/latest/develop/west/why.html# https://openwrt.org/docs/guide-developer/feeds All solve a similar problem. Part of them even overlap already with functionality from our waf based build system. We should consider whether we need any of the functionality of these tools and whether we want to re-implement it or whether we just want to use one of the tools and integrate with it. Oh, do I sense starting academic debate now? The point of my dirty PoC was to ground that a bit and show real stuff to get a feel of possibilities. Anyway, based on my PoC experience I would probably prefer something more KISS and also more flexible by tool(s) being also aware of RTEMS BSPs - first lets start with just 'boot' and 'update' scripts hosted in the repo and used to clone/update externals would be needed. And put 'external' into git ignore. - later boot/update my be made BSP aware. I mean when doing development on H7 I do not need HALs for Xilinx and other ARM/v8/Rx stuff. Neither I need HAL for NXP/Microchip chips. - may this boot/update functionality be made part of ./waf tooling? Perhaps it may So based on this I can see how Zephyr gravitated to west... I've also completely removed HAL code from the BSP directory and also CMSIS v4 files from bsps/arm/include to verify compilation is really using submodules. I think the CMSIS headers are used by some other BSPs too. So that's fine for a prototype. But for the final solution, we need something different. The PoC needs to prove some things hence removal of CMSIS from the tree. Nothing more. It's a "gain experience" vehicle, not a final solution. And btw, you will not be able to find final solution without touching/testing stuff and getting your hands dirty with it. Everything has its own price: (i) adaptation/support of west for RTEMS or (ii) writing homegrown script(s)
Re: Discussion: How to handle HALs, SDKs and libraries
Hello Karel, thanks for trying that. I wouldn't object to add it to 6 but I wouldn't expect it either because it means a lot of adaptions to the release scripts. Chris most likely can tell you more about whether it's even thinkable at that point in the release cycle or not. Am 15.07.23 um 01:48 schrieb Karel Gardas: Hello, I've created setup where I've put updated STM32H7 HAL consisting of two submodules: - stm32h7xx_hal_driver - cmsis_device_h7 and ARM's - CMSIS_5 into hals/arm/stm32h7 to follow related BSP code location. I think I would prefer "contrib/hals", "external/hals" or some other more generic top directory name. It would allow to use that for other parts like for example zlib in "contrib/libs/zlib" in the future (if we agree on having libs like that there too). At the moment, you have absolute URLs for your submodules: url = https://github.com/ARM-software/CMSIS_5.git url = https://github.com/karelfv/stm32h7xx_hal_driver.git url = https://github.com/karelfv/cmsis_device_h7.git If we use submodules, I would go for relative paths and clone the repositories to our git.rtems.org and to all mirrors. It has the advantage that even if some upstream URL changes, we won't be affected. I like submodules because they are well-supported by the usual tools. Nonetheless, we should take a look at some of the reasons why other projects decided against them. For example Google has the "repo" tool used in Android. Zephyr has the "west" tool. OpenWRT uses "feeds" for something similar. I'm sure there are more. https://gerrit.googlesource.com/git-repo/ https://docs.zephyrproject.org/latest/develop/west/why.html# https://openwrt.org/docs/guide-developer/feeds All solve a similar problem. Part of them even overlap already with functionality from our waf based build system. We should consider whether we need any of the functionality of these tools and whether we want to re-implement it or whether we just want to use one of the tools and integrate with it. I've also completely removed HAL code from the BSP directory and also CMSIS v4 files from bsps/arm/include to verify compilation is really using submodules. I think the CMSIS headers are used by some other BSPs too. So that's fine for a prototype. But for the final solution, we need something different. If you like to see how this is looking, just: $ git clone https://github.com/karelfv/rtems.git rtems $ cd rtems $ git checkout rtems-stm32h7-hal-git-submodules $ git submodule update --init And see bsps/arm/stm32h7/ and hals/arm/stm32h7/ and bsps/arm/include subdirectories. You can also compile if you have arm tools: $ ./waf bspdefaults --rtems-bsps=arm/stm32h757i-eval > stm32h757i-eval.ini $ ./waf configure --rtems-bsps=arm/stm32h757i-eval --rtems-config=./stm32h757i-eval.ini --rtems-tools= --prefix= $ ./waf CAVEAT: only STM32H757i-eval BSP is supported and only compilation. Installation is not supported yet. Note: HAL is updated to latest STMicro code and all related changes done in the past by Sebastian and Christian are merged into submodules projects except Doxygen tag change done by Sebastian. This is to be discussed topic if doxygen changes belongs to submodule or are not needed anymore... I agree that the doxygen shouldn't be necessary any more. But we should make sure that our Doxyfile doesn't pick up the submodules. So that would need some of the EXCLUDE* options set to the path of the submodules. Best regards Christian hello.exe/ticker.exe/paranoia.exe runs fine. Comments welcome! Karel On 7/14/23 13:53, Karel Gardas wrote: Hello, are we really that close to RTEMS 6 release that none of this is acceptable to do now? Asking since I'd also like to update stm32h7 HAL. I may do that manually or I may do that submodule way which may perhaps save some of the manual work involved as some of the patches may not be needed anymore. It depends on resolution to following questions: - stm32h7 hal contains Sebastian's modification/additions of Doxygen tags. Do we need to preserve that in the submodule? I guess not, but I'm not sure and this question of Doxygen markups was not solved in this thread yet AFAIK. - all ARMs HALs probably also depends on particular version of CMSIS files, so I guess we should use separate submodules of CMSIS per HAL. At least STM recommends that: https://htmlpreview.github.io/?https://github.com/STMicroelectronics/STM32CubeH7/blob/master/Release_Notes.html -- this would certainly be improvement in comparison with current state of the art besides update of CMSIS alone we would also support manufacture tested CMSIS version for their HAL... - where to put submodules? What is preferred location? For stm32h7 case! (i) hals/arm/stm32h7 or (ii) deps/arm/stm32h7 or (iii) contrib/arm/stm32h7 or (iv) anything else preferred? I may do stm32h7 hal + its CMSIS dependency now (or next week), but only if the chance
