Re: [PATCH 2/5] build: Use CSafeLoader if available
On Tue, May 23, 2023 at 12:04 AM Sebastian Huber wrote: > > > > On 23.05.23 02:42, Chris Johns wrote: > > On 22/5/2023 6:29 pm, Sebastian Huber wrote: > >> On 22.05.23 01:48, Chris Johns wrote: > The support for the CSafeLoader is just about 60 lines of additional > code. > It would be a nice improvement for systems supporting this feature. I > don't have > time to work on adding PyYAML with libyaml to the RTEMS Tools right now > and I > think this would open another maintenance issues. If someone would be > unable to > get a yaml module with a CSafeLoader, then he would be no longer able to > build > RTEMS. > >>> It is not about the size of the implementation, it is about making 2 > >>> classes of > >>> performance, those with and those without csafe. I am a firm believer all > >>> developers and especially core developers need to be using the exact same > >>> tools > >>> and code our community users are using. We did not always do this and it > >>> resulted in the developers being disconnected from the user experience > >>> and I > >>> decided I would do what I could to avoid this happening again. > >> > >> I really don't see the issue here. It is just the way to load the items > >> from the > >> file system. This is a very isolated task in the build and having two ways > >> to do > >> this is not a big deal from my point of view. > > > > Great, lets not go down this path and we all use the same process. > > The CSafeLoader approach makes my work more efficient, so I will > definitely use it. > > > > >>> In regards to PyYAML there are a few basic issues with it that I am not > >>> sure > >>> about. The package can be built without support for the C library even if > >>> it is > >>> installed. I cannot tell if a pip installed version is using the C YAML > >>> package > >>> or not. I would like more certainty across our supported hosts before > >>> agreeing > >>> to us using it. > >> > >> I think it is just too complicated to make sure that the user has a PyYAML > >> with > >> the CSafeLoader available when RTEMS is built. A more robust approach is a > >> fall > >> back to the current implementation if needed. > > Without a reliable way to ensure all users can equivalently use the tools, I see the topic as not worth fighting a lot about. I have seen projects that brought in build/configure time optional packages for performance tuning and optimization with fallbacks, e.g., tcmalloc instead of malloc. In those cases, it becomes harder to help debug problems since it introduces another branching point to determine what might go wrong. Although I'm sure this can be resolved given time, it doesn't sound like anyone has time currently to integrate the CSafeLoader in a way that makes certain that users get/use what we expect. I would suggest we can revisit the topic later, if Sebastian is satisfied he can carry his own local optimization around for now. I think we have probably all done things like that in the past. I appreciate the discussion and that the support to use it was brought, but unless the build times become a general problem for more developers and the user base, I don't find this debate fruitful. Gedare > > Maybe we move away from YAML for the build system data? Amar and I > > originally > > had python and INI files. > > This discussion started with a proposal to switch to JSON: > > https://lists.rtems.org/pipermail/devel/2023-April/075082.html > > The INI format is a simple key-value format with no support for > hierarchical data, so not suitable from my point of view. > > Using Python code could be an option, however, this gives the writer of > the files more freedom and room for creativity. On lesson learnt from > converting the autoconf/automake build to the new build system is, that > this may cause some issues. > > -- > embedded brains GmbH > Herr Sebastian HUBER > Dornierstr. 4 > 82178 Puchheim > Germany > email: sebastian.hu...@embedded-brains.de > phone: +49-89-18 94 741 - 16 > fax: +49-89-18 94 741 - 08 > > Registergericht: Amtsgericht München > Registernummer: HRB 157899 > Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler > Unsere Datenschutzerklärung finden Sie hier: > https://embedded-brains.de/datenschutzerklaerung/ ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 2/5] build: Use CSafeLoader if available
On 23.05.23 02:42, Chris Johns wrote: On 22/5/2023 6:29 pm, Sebastian Huber wrote: On 22.05.23 01:48, Chris Johns wrote: The support for the CSafeLoader is just about 60 lines of additional code. It would be a nice improvement for systems supporting this feature. I don't have time to work on adding PyYAML with libyaml to the RTEMS Tools right now and I think this would open another maintenance issues. If someone would be unable to get a yaml module with a CSafeLoader, then he would be no longer able to build RTEMS. It is not about the size of the implementation, it is about making 2 classes of performance, those with and those without csafe. I am a firm believer all developers and especially core developers need to be using the exact same tools and code our community users are using. We did not always do this and it resulted in the developers being disconnected from the user experience and I decided I would do what I could to avoid this happening again. I really don't see the issue here. It is just the way to load the items from the file system. This is a very isolated task in the build and having two ways to do this is not a big deal from my point of view. Great, lets not go down this path and we all use the same process. The CSafeLoader approach makes my work more efficient, so I will definitely use it. In regards to PyYAML there are a few basic issues with it that I am not sure about. The package can be built without support for the C library even if it is installed. I cannot tell if a pip installed version is using the C YAML package or not. I would like more certainty across our supported hosts before agreeing to us using it. I think it is just too complicated to make sure that the user has a PyYAML with the CSafeLoader available when RTEMS is built. A more robust approach is a fall back to the current implementation if needed. Maybe we move away from YAML for the build system data? Amar and I originally had python and INI files. This discussion started with a proposal to switch to JSON: https://lists.rtems.org/pipermail/devel/2023-April/075082.html The INI format is a simple key-value format with no support for hierarchical data, so not suitable from my point of view. Using Python code could be an option, however, this gives the writer of the files more freedom and room for creativity. On lesson learnt from converting the autoconf/automake build to the new build system is, that this may cause some issues. -- embedded brains GmbH Herr Sebastian HUBER Dornierstr. 4 82178 Puchheim Germany email: sebastian.hu...@embedded-brains.de phone: +49-89-18 94 741 - 16 fax: +49-89-18 94 741 - 08 Registergericht: Amtsgericht München Registernummer: HRB 157899 Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler Unsere Datenschutzerklärung finden Sie hier: https://embedded-brains.de/datenschutzerklaerung/ ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 2/5] build: Use CSafeLoader if available
On 22/5/2023 6:29 pm, Sebastian Huber wrote: > On 22.05.23 01:48, Chris Johns wrote: >>> The support for the CSafeLoader is just about 60 lines of additional code. >>> It would be a nice improvement for systems supporting this feature. I don't >>> have >>> time to work on adding PyYAML with libyaml to the RTEMS Tools right now and >>> I >>> think this would open another maintenance issues. If someone would be >>> unable to >>> get a yaml module with a CSafeLoader, then he would be no longer able to >>> build >>> RTEMS. >> It is not about the size of the implementation, it is about making 2 classes >> of >> performance, those with and those without csafe. I am a firm believer all >> developers and especially core developers need to be using the exact same >> tools >> and code our community users are using. We did not always do this and it >> resulted in the developers being disconnected from the user experience and I >> decided I would do what I could to avoid this happening again. > > I really don't see the issue here. It is just the way to load the items from > the > file system. This is a very isolated task in the build and having two ways to > do > this is not a big deal from my point of view. Great, lets not go down this path and we all use the same process. >> In regards to PyYAML there are a few basic issues with it that I am not sure >> about. The package can be built without support for the C library even if it >> is >> installed. I cannot tell if a pip installed version is using the C YAML >> package >> or not. I would like more certainty across our supported hosts before >> agreeing >> to us using it. > > I think it is just too complicated to make sure that the user has a PyYAML > with > the CSafeLoader available when RTEMS is built. A more robust approach is a > fall > back to the current implementation if needed. Maybe we move away from YAML for the build system data? Amar and I originally had python and INI files. Chris ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 2/5] build: Use CSafeLoader if available
On 22.05.23 01:48, Chris Johns wrote: The support for the CSafeLoader is just about 60 lines of additional code. It would be a nice improvement for systems supporting this feature. I don't have time to work on adding PyYAML with libyaml to the RTEMS Tools right now and I think this would open another maintenance issues. If someone would be unable to get a yaml module with a CSafeLoader, then he would be no longer able to build RTEMS. It is not about the size of the implementation, it is about making 2 classes of performance, those with and those without csafe. I am a firm believer all developers and especially core developers need to be using the exact same tools and code our community users are using. We did not always do this and it resulted in the developers being disconnected from the user experience and I decided I would do what I could to avoid this happening again. I really don't see the issue here. It is just the way to load the items from the file system. This is a very isolated task in the build and having two ways to do this is not a big deal from my point of view. In regards to PyYAML there are a few basic issues with it that I am not sure about. The package can be built without support for the C library even if it is installed. I cannot tell if a pip installed version is using the C YAML package or not. I would like more certainty across our supported hosts before agreeing to us using it. I think it is just too complicated to make sure that the user has a PyYAML with the CSafeLoader available when RTEMS is built. A more robust approach is a fall back to the current implementation if needed. -- embedded brains GmbH Herr Sebastian HUBER Dornierstr. 4 82178 Puchheim Germany email: sebastian.hu...@embedded-brains.de phone: +49-89-18 94 741 - 16 fax: +49-89-18 94 741 - 08 Registergericht: Amtsgericht München Registernummer: HRB 157899 Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler Unsere Datenschutzerklärung finden Sie hier: https://embedded-brains.de/datenschutzerklaerung/ ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 2/5] build: Use CSafeLoader if available
On 19/5/2023 4:46 pm, Sebastian Huber wrote: > On 15.05.23 01:44, Chris Johns wrote: >> On 5/5/2023 8:16 pm, Sebastian Huber wrote: >>> On 05.05.23 01:38, Chris Johns wrote: > When I install it > through pip, I get the CSafeLoader on my machine. I don't have a libyaml > development package installed. This is what I was not understanding. If it can be loaded in a python.org install on a N2 MacOS with a virtual env then I think we are close to a resolution. >>> I tried to understand how the PyYAML package is built. It seems that >>> libyaml and >>> its header files need to be installed to use the C bindings. >> This is what I thought happened with this package. It depends on the base OS >> packaging system. The documentation at >> https://pyyaml.org/wiki/PyYAMLDocumentation says: >> >> $ python setup.py install >> >> If you want to use LibYAML bindings, which are much faster than the pure >> Python version, you need to download and install LibYAML. Then you may >> build and install the bindings by executing >> >> $ python setup.py --with-libyaml install >> >> I cannot see if pip has any further detail on the nature of the build it >> contains so this is not looking good. This leaves building and installing >> when >> the tools are built by the RSB. > > The current approach with the pickle cache works reasonably fine and requires > not extra packages. I don't think this implementation should be removed in any > case. Agreed. > The support for the CSafeLoader is just about 60 lines of additional code. > It would be a nice improvement for systems supporting this feature. I don't > have > time to work on adding PyYAML with libyaml to the RTEMS Tools right now and I > think this would open another maintenance issues. If someone would be unable > to > get a yaml module with a CSafeLoader, then he would be no longer able to build > RTEMS. It is not about the size of the implementation, it is about making 2 classes of performance, those with and those without csafe. I am a firm believer all developers and especially core developers need to be using the exact same tools and code our community users are using. We did not always do this and it resulted in the developers being disconnected from the user experience and I decided I would do what I could to avoid this happening again. In regards to PyYAML there are a few basic issues with it that I am not sure about. The package can be built without support for the C library even if it is installed. I cannot tell if a pip installed version is using the C YAML package or not. I would like more certainty across our supported hosts before agreeing to us using it. Chris ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 2/5] build: Use CSafeLoader if available
On 15.05.23 01:44, Chris Johns wrote: On 5/5/2023 8:16 pm, Sebastian Huber wrote: On 05.05.23 01:38, Chris Johns wrote: When I install it through pip, I get the CSafeLoader on my machine. I don't have a libyaml development package installed. This is what I was not understanding. If it can be loaded in a python.org install on a N2 MacOS with a virtual env then I think we are close to a resolution. I tried to understand how the PyYAML package is built. It seems that libyaml and its header files need to be installed to use the C bindings. This is what I thought happened with this package. It depends on the base OS packaging system. The documentation at https://pyyaml.org/wiki/PyYAMLDocumentation says: $ python setup.py install If you want to use LibYAML bindings, which are much faster than the pure Python version, you need to download and install LibYAML. Then you may build and install the bindings by executing $ python setup.py --with-libyaml install I cannot see if pip has any further detail on the nature of the build it contains so this is not looking good. This leaves building and installing when the tools are built by the RSB. The current approach with the pickle cache works reasonably fine and requires not extra packages. I don't think this implementation should be removed in any case. The support for the CSafeLoader is just about 60 lines of additional code. It would be a nice improvement for systems supporting this feature. I don't have time to work on adding PyYAML with libyaml to the RTEMS Tools right now and I think this would open another maintenance issues. If someone would be unable to get a yaml module with a CSafeLoader, then he would be no longer able to build RTEMS. -- embedded brains GmbH Herr Sebastian HUBER Dornierstr. 4 82178 Puchheim Germany email: sebastian.hu...@embedded-brains.de phone: +49-89-18 94 741 - 16 fax: +49-89-18 94 741 - 08 Registergericht: Amtsgericht München Registernummer: HRB 157899 Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler Unsere Datenschutzerklärung finden Sie hier: https://embedded-brains.de/datenschutzerklaerung/ ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 2/5] build: Use CSafeLoader if available
On 5/5/2023 8:16 pm, Sebastian Huber wrote: > On 05.05.23 01:38, Chris Johns wrote: >>> When I install it >>> through pip, I get the CSafeLoader on my machine. I don't have a libyaml >>> development package installed. >> This is what I was not understanding. If it can be loaded in a python.org >> install on a N2 MacOS with a virtual env then I think we are close to a >> resolution. > > I tried to understand how the PyYAML package is built. It seems that libyaml > and > its header files need to be installed to use the C bindings. This is what I thought happened with this package. It depends on the base OS packaging system. The documentation at https://pyyaml.org/wiki/PyYAMLDocumentation says: $ python setup.py install If you want to use LibYAML bindings, which are much faster than the pure Python version, you need to download and install LibYAML. Then you may build and install the bindings by executing $ python setup.py --with-libyaml install I cannot see if pip has any further detail on the nature of the build it contains so this is not looking good. This leaves building and installing when the tools are built by the RSB. Chris ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 2/5] build: Use CSafeLoader if available
On 05.05.23 01:38, Chris Johns wrote: When I install it through pip, I get the CSafeLoader on my machine. I don't have a libyaml development package installed. This is what I was not understanding. If it can be loaded in a python.org install on a N2 MacOS with a virtual env then I think we are close to a resolution. I tried to understand how the PyYAML package is built. It seems that libyaml and its header files need to be installed to use the C bindings. -- embedded brains GmbH Herr Sebastian HUBER Dornierstr. 4 82178 Puchheim Germany email: sebastian.hu...@embedded-brains.de phone: +49-89-18 94 741 - 16 fax: +49-89-18 94 741 - 08 Registergericht: Amtsgericht München Registernummer: HRB 157899 Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler Unsere Datenschutzerklärung finden Sie hier: https://embedded-brains.de/datenschutzerklaerung/ ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 2/5] build: Use CSafeLoader if available
On 4/5/2023 4:16 pm, Sebastian Huber wrote: > On 04.05.23 05:35, Chris Johns wrote: >> On 3/5/2023 7:40 pm, Sebastian Huber wrote: >>> On 03.05.23 05:30, Chris Johns wrote: On 28/4/2023 3:38 pm, Sebastian Huber wrote: > On 27.04.23 20:27, Gedare Bloom wrote: >> On Wed, Apr 26, 2023 at 11:46 PM Sebastian Huber >> wrote: >>> On 27.04.23 02:11, Chris Johns wrote: On 26/4/2023 6:04 pm, Sebastian Huber wrote: > The CSafeLoader uses the C libyaml libary to considerably speed up the > loading of YAML files. No from me. >>> What do you mean with not for me? You have the CSafeLoader available and >>> it is slow? Do you have some timings before and after the patch set for >>> a "./waf configure" and "./waf build"? On my systems the configure needs >>> less than a second with the CSafeLoader and the waf build setup time is >>> less than 100ms. >>> I do not agree with conditional states of operation in the build system that depend on packages a host has installed. If speed is an important factor all users then I suggest you find a means to have it available automatically on the hosts we support (Linux, FreeBSD, MacOS, Windows MINGW64 and Cygwin. >>> I am not sure if we should automatically install system Python packages >>> on user machines. >>> >>> The fall back is the Python PyYAML package available through the RTEMS >>> sources. This is what we use currently. For RTEMS users, this is >>> acceptable since they are not supposed to touch the YAML files. For >>> RTEMS maintainers, not having the cache makes working with the build >>> system more efficient. >>> >>> If they system PyYAML package is not installed, then you get now a hint >>> to install it: >>> >>> Setting top to : >>> /home/EB/sebastian_h/src/rtems >>> Setting out to : >>> /home/EB/sebastian_h/src/rtems/build >>> Regenerate the build specification cache. Install the PyYAML Python >>> package to avoid this. The cache regeneration needs a couple of >>> seconds... >>> Configure board support package (BSP) : arm/realview_pbx_a9_qemu >>> >> I have two questions, which are related to Chris's concern I think. >> 1. Are the output of PyYAML and C libyaml guaranteed to be consistent? > > I trust the PyYAML maintainers that the SafeLoader and CSafeLoader > produce the > same results. With respect to the alternative ItemCache class > implementation in > the wscript I am quite confident that this produces the same results. This > part > just has to load the item data from the files. The CSafeLoader based > ItemCache > has 53 lines of code. > >> >> 2. Why not make C libyaml part of the RTEMS toolchain? >> >> Any dependencies that exist in the build system are (by definition) >> suitable to be checked/provided by the tool buildset. > > Yes, this is an option. If we remove the pickle cache, then we force > everyone to > use the libyaml based PyYAML module. Is this really necessary right now? If we leave it who would do it? I would like to understand the next question before we decide if this is important. The key objective is to have consistent performance for every one. If the package is easy to build then we should do it when we build the tools and the questions we are having go away. >>> >>> The PyYAML package had some security issues in the past. If we ship this >>> package, who will monitor this package, update it, and write security >>> advisories? >> >> The same way we would handle any security issue. When we become aware we >> update >> what we provide. > > This is a problem from my point of view. Maintenance activities (including > security related topics) happen by accident in the RTEMS Project. In general, > each mandatory host tool makes it harder to install RTEMS in certain > environments. > >> >> Is PyYAML a pip package or is it provided by a distro package when using >> Linux? >> My assumption, which may be wrong, is building libyaml (the C part) is all we >> need to do? > > You can install it through pip, conda, or whatever your host provides as > packages. I guess you need to build also some Python bindings for libyaml to > be > able to use it. Using pip with a virtual env is the path I think we should document for users. With python3 is it easy to do and safe because the packages are contained and localised to the RTEMS environment. Other solution can be used for those who are across python and the packages so they can manage themselves. When I say safe I mean the results are controlled and we are able to provide support if something is not working as it should. > For > most use cases the Python only
Re: [PATCH 2/5] build: Use CSafeLoader if available
On 04.05.23 05:35, Chris Johns wrote: On 3/5/2023 7:40 pm, Sebastian Huber wrote: On 03.05.23 05:30, Chris Johns wrote: On 28/4/2023 3:38 pm, Sebastian Huber wrote: On 27.04.23 20:27, Gedare Bloom wrote: On Wed, Apr 26, 2023 at 11:46 PM Sebastian Huber wrote: On 27.04.23 02:11, Chris Johns wrote: On 26/4/2023 6:04 pm, Sebastian Huber wrote: The CSafeLoader uses the C libyaml libary to considerably speed up the loading of YAML files. No from me. What do you mean with not for me? You have the CSafeLoader available and it is slow? Do you have some timings before and after the patch set for a "./waf configure" and "./waf build"? On my systems the configure needs less than a second with the CSafeLoader and the waf build setup time is less than 100ms. I do not agree with conditional states of operation in the build system that depend on packages a host has installed. If speed is an important factor all users then I suggest you find a means to have it available automatically on the hosts we support (Linux, FreeBSD, MacOS, Windows MINGW64 and Cygwin. I am not sure if we should automatically install system Python packages on user machines. The fall back is the Python PyYAML package available through the RTEMS sources. This is what we use currently. For RTEMS users, this is acceptable since they are not supposed to touch the YAML files. For RTEMS maintainers, not having the cache makes working with the build system more efficient. If they system PyYAML package is not installed, then you get now a hint to install it: Setting top to : /home/EB/sebastian_h/src/rtems Setting out to : /home/EB/sebastian_h/src/rtems/build Regenerate the build specification cache. Install the PyYAML Python package to avoid this. The cache regeneration needs a couple of seconds... Configure board support package (BSP) : arm/realview_pbx_a9_qemu I have two questions, which are related to Chris's concern I think. 1. Are the output of PyYAML and C libyaml guaranteed to be consistent? I trust the PyYAML maintainers that the SafeLoader and CSafeLoader produce the same results. With respect to the alternative ItemCache class implementation in the wscript I am quite confident that this produces the same results. This part just has to load the item data from the files. The CSafeLoader based ItemCache has 53 lines of code. 2. Why not make C libyaml part of the RTEMS toolchain? Any dependencies that exist in the build system are (by definition) suitable to be checked/provided by the tool buildset. Yes, this is an option. If we remove the pickle cache, then we force everyone to use the libyaml based PyYAML module. Is this really necessary right now? If we leave it who would do it? I would like to understand the next question before we decide if this is important. The key objective is to have consistent performance for every one. If the package is easy to build then we should do it when we build the tools and the questions we are having go away. The PyYAML package had some security issues in the past. If we ship this package, who will monitor this package, update it, and write security advisories? The same way we would handle any security issue. When we become aware we update what we provide. This is a problem from my point of view. Maintenance activities (including security related topics) happen by accident in the RTEMS Project. In general, each mandatory host tool makes it harder to install RTEMS in certain environments. Is PyYAML a pip package or is it provided by a distro package when using Linux? My assumption, which may be wrong, is building libyaml (the C part) is all we need to do? You can install it through pip, conda, or whatever your host provides as packages. I guess you need to build also some Python bindings for libyaml to be able to use it. For most use cases the Python only solution works fine. If you spend your time developing BSPs, then the CSafeLoader pays off. Maybe I am not understanding how this works. Why is there a difference for developers vs a user who does not have this package installed? Does the difference scale? A user typically just uses a certain version of RTEMS. Then the BSPs of interest are configured and built. A user is not supposed to touch the spec files. My experience is different. I do not agree with different levels of performance and build experience based on the host operating system being used. We need to support all hosts in the same way and this seems to favour users who have an OS that can provide the package. We have had host biases other places in RTEMS and it takes a long time to remove it. The policy I work to is RTEMS developers and users use the same tools and processes and this has been working well through my time with this project. I see no reason to move away from this. I don't see the problem here, PyYAML is a widely used package. When I install it through pip, I get the
Re: [PATCH 2/5] build: Use CSafeLoader if available
On 3/5/2023 7:40 pm, Sebastian Huber wrote: > On 03.05.23 05:30, Chris Johns wrote: >> On 28/4/2023 3:38 pm, Sebastian Huber wrote: >>> On 27.04.23 20:27, Gedare Bloom wrote: On Wed, Apr 26, 2023 at 11:46 PM Sebastian Huber wrote: > On 27.04.23 02:11, Chris Johns wrote: >> On 26/4/2023 6:04 pm, Sebastian Huber wrote: >>> The CSafeLoader uses the C libyaml libary to considerably speed up the >>> loading of YAML files. >> No from me. > What do you mean with not for me? You have the CSafeLoader available and > it is slow? Do you have some timings before and after the patch set for > a "./waf configure" and "./waf build"? On my systems the configure needs > less than a second with the CSafeLoader and the waf build setup time is > less than 100ms. > >> I do not agree with conditional states of operation in the build system >> that >> depend on packages a host has installed. If speed is an important factor >> all >> users then I suggest you find a means to have it available automatically >> on the >> hosts we support (Linux, FreeBSD, MacOS, Windows MINGW64 and Cygwin. > I am not sure if we should automatically install system Python packages > on user machines. > > The fall back is the Python PyYAML package available through the RTEMS > sources. This is what we use currently. For RTEMS users, this is > acceptable since they are not supposed to touch the YAML files. For > RTEMS maintainers, not having the cache makes working with the build > system more efficient. > > If they system PyYAML package is not installed, then you get now a hint > to install it: > > Setting top to : /home/EB/sebastian_h/src/rtems > Setting out to : > /home/EB/sebastian_h/src/rtems/build > Regenerate the build specification cache. Install the PyYAML Python > package to avoid this. The cache regeneration needs a couple of > seconds... > Configure board support package (BSP) : arm/realview_pbx_a9_qemu > I have two questions, which are related to Chris's concern I think. 1. Are the output of PyYAML and C libyaml guaranteed to be consistent? >>> >>> I trust the PyYAML maintainers that the SafeLoader and CSafeLoader produce >>> the >>> same results. With respect to the alternative ItemCache class >>> implementation in >>> the wscript I am quite confident that this produces the same results. This >>> part >>> just has to load the item data from the files. The CSafeLoader based >>> ItemCache >>> has 53 lines of code. >>> 2. Why not make C libyaml part of the RTEMS toolchain? Any dependencies that exist in the build system are (by definition) suitable to be checked/provided by the tool buildset. >>> >>> Yes, this is an option. If we remove the pickle cache, then we force >>> everyone to >>> use the libyaml based PyYAML module. Is this really necessary right now? >> >> If we leave it who would do it? I would like to understand the next question >> before we decide if this is important. The key objective is to have >> consistent >> performance for every one. If the package is easy to build then we should do >> it >> when we build the tools and the questions we are having go away. > > The PyYAML package had some security issues in the past. If we ship this > package, who will monitor this package, update it, and write security > advisories? The same way we would handle any security issue. When we become aware we update what we provide. Is PyYAML a pip package or is it provided by a distro package when using Linux? My assumption, which may be wrong, is building libyaml (the C part) is all we need to do? >>> For >>> most use cases the Python only solution works fine. If you spend your time >>> developing BSPs, then the CSafeLoader pays off. >> >> Maybe I am not understanding how this works. Why is there a difference for >> developers vs a user who does not have this package installed? Does the >> difference scale? > > A user typically just uses a certain version of RTEMS. Then the BSPs of > interest > are configured and built. A user is not supposed to touch the spec files. My experience is different. I do not agree with different levels of performance and build experience based on the host operating system being used. We need to support all hosts in the same way and this seems to favour users who have an OS that can provide the package. We have had host biases other places in RTEMS and it takes a long time to remove it. The policy I work to is RTEMS developers and users use the same tools and processes and this has been working well through my time with this project. I see no reason to move away from this. > A maintainer adds, modifies, removes spec files during development. With the > item cache, this always involves a time to wait of several seconds. the time > to > wait
Re: [PATCH 2/5] build: Use CSafeLoader if available
On 03.05.23 05:30, Chris Johns wrote: On 28/4/2023 3:38 pm, Sebastian Huber wrote: On 27.04.23 20:27, Gedare Bloom wrote: On Wed, Apr 26, 2023 at 11:46 PM Sebastian Huber wrote: On 27.04.23 02:11, Chris Johns wrote: On 26/4/2023 6:04 pm, Sebastian Huber wrote: The CSafeLoader uses the C libyaml libary to considerably speed up the loading of YAML files. No from me. What do you mean with not for me? You have the CSafeLoader available and it is slow? Do you have some timings before and after the patch set for a "./waf configure" and "./waf build"? On my systems the configure needs less than a second with the CSafeLoader and the waf build setup time is less than 100ms. I do not agree with conditional states of operation in the build system that depend on packages a host has installed. If speed is an important factor all users then I suggest you find a means to have it available automatically on the hosts we support (Linux, FreeBSD, MacOS, Windows MINGW64 and Cygwin. I am not sure if we should automatically install system Python packages on user machines. The fall back is the Python PyYAML package available through the RTEMS sources. This is what we use currently. For RTEMS users, this is acceptable since they are not supposed to touch the YAML files. For RTEMS maintainers, not having the cache makes working with the build system more efficient. If they system PyYAML package is not installed, then you get now a hint to install it: Setting top to : /home/EB/sebastian_h/src/rtems Setting out to : /home/EB/sebastian_h/src/rtems/build Regenerate the build specification cache. Install the PyYAML Python package to avoid this. The cache regeneration needs a couple of seconds... Configure board support package (BSP) : arm/realview_pbx_a9_qemu I have two questions, which are related to Chris's concern I think. 1. Are the output of PyYAML and C libyaml guaranteed to be consistent? I trust the PyYAML maintainers that the SafeLoader and CSafeLoader produce the same results. With respect to the alternative ItemCache class implementation in the wscript I am quite confident that this produces the same results. This part just has to load the item data from the files. The CSafeLoader based ItemCache has 53 lines of code. 2. Why not make C libyaml part of the RTEMS toolchain? Any dependencies that exist in the build system are (by definition) suitable to be checked/provided by the tool buildset. Yes, this is an option. If we remove the pickle cache, then we force everyone to use the libyaml based PyYAML module. Is this really necessary right now? If we leave it who would do it? I would like to understand the next question before we decide if this is important. The key objective is to have consistent performance for every one. If the package is easy to build then we should do it when we build the tools and the questions we are having go away. The PyYAML package had some security issues in the past. If we ship this package, who will monitor this package, update it, and write security advisories? For most use cases the Python only solution works fine. If you spend your time developing BSPs, then the CSafeLoader pays off. Maybe I am not understanding how this works. Why is there a difference for developers vs a user who does not have this package installed? Does the difference scale? A user typically just uses a certain version of RTEMS. Then the BSPs of interest are configured and built. A user is not supposed to touch the spec files. A maintainer adds, modifies, removes spec files during development. With the item cache, this always involves a time to wait of several seconds. the time to wait depends on the total number of spec files. With the CSafeLoader this time is reduced to a fraction of a second. -- embedded brains GmbH Herr Sebastian HUBER Dornierstr. 4 82178 Puchheim Germany email: sebastian.hu...@embedded-brains.de phone: +49-89-18 94 741 - 16 fax: +49-89-18 94 741 - 08 Registergericht: Amtsgericht München Registernummer: HRB 157899 Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler Unsere Datenschutzerklärung finden Sie hier: https://embedded-brains.de/datenschutzerklaerung/ ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 2/5] build: Use CSafeLoader if available
On 28/4/2023 3:38 pm, Sebastian Huber wrote: > On 27.04.23 20:27, Gedare Bloom wrote: >> On Wed, Apr 26, 2023 at 11:46 PM Sebastian Huber >> wrote: >>> On 27.04.23 02:11, Chris Johns wrote: On 26/4/2023 6:04 pm, Sebastian Huber wrote: > The CSafeLoader uses the C libyaml libary to considerably speed up the > loading of YAML files. No from me. >>> What do you mean with not for me? You have the CSafeLoader available and >>> it is slow? Do you have some timings before and after the patch set for >>> a "./waf configure" and "./waf build"? On my systems the configure needs >>> less than a second with the CSafeLoader and the waf build setup time is >>> less than 100ms. >>> I do not agree with conditional states of operation in the build system that depend on packages a host has installed. If speed is an important factor all users then I suggest you find a means to have it available automatically on the hosts we support (Linux, FreeBSD, MacOS, Windows MINGW64 and Cygwin. >>> I am not sure if we should automatically install system Python packages >>> on user machines. >>> >>> The fall back is the Python PyYAML package available through the RTEMS >>> sources. This is what we use currently. For RTEMS users, this is >>> acceptable since they are not supposed to touch the YAML files. For >>> RTEMS maintainers, not having the cache makes working with the build >>> system more efficient. >>> >>> If they system PyYAML package is not installed, then you get now a hint >>> to install it: >>> >>> Setting top to : /home/EB/sebastian_h/src/rtems >>> Setting out to : >>> /home/EB/sebastian_h/src/rtems/build >>> Regenerate the build specification cache. Install the PyYAML Python >>> package to avoid this. The cache regeneration needs a couple of seconds... >>> Configure board support package (BSP) : arm/realview_pbx_a9_qemu >>> >> I have two questions, which are related to Chris's concern I think. >> 1. Are the output of PyYAML and C libyaml guaranteed to be consistent? > > I trust the PyYAML maintainers that the SafeLoader and CSafeLoader produce the > same results. With respect to the alternative ItemCache class implementation > in > the wscript I am quite confident that this produces the same results. This > part > just has to load the item data from the files. The CSafeLoader based ItemCache > has 53 lines of code. > >> >> 2. Why not make C libyaml part of the RTEMS toolchain? >> >> Any dependencies that exist in the build system are (by definition) >> suitable to be checked/provided by the tool buildset. > > Yes, this is an option. If we remove the pickle cache, then we force everyone > to > use the libyaml based PyYAML module. Is this really necessary right now? If we leave it who would do it? I would like to understand the next question before we decide if this is important. The key objective is to have consistent performance for every one. If the package is easy to build then we should do it when we build the tools and the questions we are having go away. > For > most use cases the Python only solution works fine. If you spend your time > developing BSPs, then the CSafeLoader pays off. Maybe I am not understanding how this works. Why is there a difference for developers vs a user who does not have this package installed? Does the difference scale? Chris ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 2/5] build: Use CSafeLoader if available
On 27.04.23 23:59, Chris Johns wrote: On 28/4/2023 4:27 am, Gedare Bloom wrote: On Wed, Apr 26, 2023 at 11:46 PM Sebastian Huber wrote: On 27.04.23 02:11, Chris Johns wrote: On 26/4/2023 6:04 pm, Sebastian Huber wrote: The CSafeLoader uses the C libyaml libary to considerably speed up the loading of YAML files. No from me. What do you mean with not for me? You have the CSafeLoader available and it is slow? Do you have some timings before and after the patch set for a "./waf configure" and "./waf build"? On my systems the configure needs less than a second with the CSafeLoader and the waf build setup time is less than 100ms. I do not agree this changing being merged as is. A move to csafeloader for all users is a good idea but not conditionally. Ok, good. I have currently no time to dig into the specifics of deploying a Python module with C extensions, so I will simply use this patch set internally. -- embedded brains GmbH Herr Sebastian HUBER Dornierstr. 4 82178 Puchheim Germany email: sebastian.hu...@embedded-brains.de phone: +49-89-18 94 741 - 16 fax: +49-89-18 94 741 - 08 Registergericht: Amtsgericht München Registernummer: HRB 157899 Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler Unsere Datenschutzerklärung finden Sie hier: https://embedded-brains.de/datenschutzerklaerung/ ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 2/5] build: Use CSafeLoader if available
On 27.04.23 20:27, Gedare Bloom wrote: On Wed, Apr 26, 2023 at 11:46 PM Sebastian Huber wrote: On 27.04.23 02:11, Chris Johns wrote: On 26/4/2023 6:04 pm, Sebastian Huber wrote: The CSafeLoader uses the C libyaml libary to considerably speed up the loading of YAML files. No from me. What do you mean with not for me? You have the CSafeLoader available and it is slow? Do you have some timings before and after the patch set for a "./waf configure" and "./waf build"? On my systems the configure needs less than a second with the CSafeLoader and the waf build setup time is less than 100ms. I do not agree with conditional states of operation in the build system that depend on packages a host has installed. If speed is an important factor all users then I suggest you find a means to have it available automatically on the hosts we support (Linux, FreeBSD, MacOS, Windows MINGW64 and Cygwin. I am not sure if we should automatically install system Python packages on user machines. The fall back is the Python PyYAML package available through the RTEMS sources. This is what we use currently. For RTEMS users, this is acceptable since they are not supposed to touch the YAML files. For RTEMS maintainers, not having the cache makes working with the build system more efficient. If they system PyYAML package is not installed, then you get now a hint to install it: Setting top to : /home/EB/sebastian_h/src/rtems Setting out to : /home/EB/sebastian_h/src/rtems/build Regenerate the build specification cache. Install the PyYAML Python package to avoid this. The cache regeneration needs a couple of seconds... Configure board support package (BSP): arm/realview_pbx_a9_qemu I have two questions, which are related to Chris's concern I think. 1. Are the output of PyYAML and C libyaml guaranteed to be consistent? I trust the PyYAML maintainers that the SafeLoader and CSafeLoader produce the same results. With respect to the alternative ItemCache class implementation in the wscript I am quite confident that this produces the same results. This part just has to load the item data from the files. The CSafeLoader based ItemCache has 53 lines of code. 2. Why not make C libyaml part of the RTEMS toolchain? Any dependencies that exist in the build system are (by definition) suitable to be checked/provided by the tool buildset. Yes, this is an option. If we remove the pickle cache, then we force everyone to use the libyaml based PyYAML module. Is this really necessary right now? For most use cases the Python only solution works fine. If you spend your time developing BSPs, then the CSafeLoader pays off. -- embedded brains GmbH Herr Sebastian HUBER Dornierstr. 4 82178 Puchheim Germany email: sebastian.hu...@embedded-brains.de phone: +49-89-18 94 741 - 16 fax: +49-89-18 94 741 - 08 Registergericht: Amtsgericht München Registernummer: HRB 157899 Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler Unsere Datenschutzerklärung finden Sie hier: https://embedded-brains.de/datenschutzerklaerung/ ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 2/5] build: Use CSafeLoader if available
On 28/4/2023 4:27 am, Gedare Bloom wrote: > On Wed, Apr 26, 2023 at 11:46 PM Sebastian Huber > wrote: >> >> On 27.04.23 02:11, Chris Johns wrote: >>> On 26/4/2023 6:04 pm, Sebastian Huber wrote: The CSafeLoader uses the C libyaml libary to considerably speed up the loading of YAML files. >>> No from me. >> >> What do you mean with not for me? You have the CSafeLoader available and >> it is slow? Do you have some timings before and after the patch set for >> a "./waf configure" and "./waf build"? On my systems the configure needs >> less than a second with the CSafeLoader and the waf build setup time is >> less than 100ms. I do not agree this changing being merged as is. A move to csafeloader for all users is a good idea but not conditionally. >>> I do not agree with conditional states of operation in the build system that >>> depend on packages a host has installed. If speed is an important factor all >>> users then I suggest you find a means to have it available automatically on >>> the >>> hosts we support (Linux, FreeBSD, MacOS, Windows MINGW64 and Cygwin. >> >> I am not sure if we should automatically install system Python packages >> on user machines. >> >> The fall back is the Python PyYAML package available through the RTEMS >> sources. This is what we use currently. For RTEMS users, this is >> acceptable since they are not supposed to touch the YAML files. For >> RTEMS maintainers, not having the cache makes working with the build >> system more efficient. >> >> If they system PyYAML package is not installed, then you get now a hint >> to install it: >> >> Setting top to : /home/EB/sebastian_h/src/rtems >> Setting out to : >> /home/EB/sebastian_h/src/rtems/build >> Regenerate the build specification cache. Install the PyYAML Python >> package to avoid this. The cache regeneration needs a couple of seconds... >> Configure board support package (BSP): arm/realview_pbx_a9_qemu >> > > I have two questions, which are related to Chris's concern I think. > 1. Are the output of PyYAML and C libyaml guaranteed to be consistent? > > 2. Why not make C libyaml part of the RTEMS toolchain? > > Any dependencies that exist in the build system are (by definition) > suitable to be checked/provided by the tool buildset. Yes this is a good summary, thanks. Chris ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 2/5] build: Use CSafeLoader if available
On Wed, Apr 26, 2023 at 11:46 PM Sebastian Huber wrote: > > On 27.04.23 02:11, Chris Johns wrote: > > On 26/4/2023 6:04 pm, Sebastian Huber wrote: > >> The CSafeLoader uses the C libyaml libary to considerably speed up the > >> loading of YAML files. > > No from me. > > What do you mean with not for me? You have the CSafeLoader available and > it is slow? Do you have some timings before and after the patch set for > a "./waf configure" and "./waf build"? On my systems the configure needs > less than a second with the CSafeLoader and the waf build setup time is > less than 100ms. > > > I do not agree with conditional states of operation in the build system that > > depend on packages a host has installed. If speed is an important factor all > > users then I suggest you find a means to have it available automatically on > > the > > hosts we support (Linux, FreeBSD, MacOS, Windows MINGW64 and Cygwin. > > I am not sure if we should automatically install system Python packages > on user machines. > > The fall back is the Python PyYAML package available through the RTEMS > sources. This is what we use currently. For RTEMS users, this is > acceptable since they are not supposed to touch the YAML files. For > RTEMS maintainers, not having the cache makes working with the build > system more efficient. > > If they system PyYAML package is not installed, then you get now a hint > to install it: > > Setting top to : /home/EB/sebastian_h/src/rtems > Setting out to : > /home/EB/sebastian_h/src/rtems/build > Regenerate the build specification cache. Install the PyYAML Python > package to avoid this. The cache regeneration needs a couple of seconds... > Configure board support package (BSP): arm/realview_pbx_a9_qemu > I have two questions, which are related to Chris's concern I think. 1. Are the output of PyYAML and C libyaml guaranteed to be consistent? 2. Why not make C libyaml part of the RTEMS toolchain? Any dependencies that exist in the build system are (by definition) suitable to be checked/provided by the tool buildset. Gedare > > -- > embedded brains GmbH > Herr Sebastian HUBER > Dornierstr. 4 > 82178 Puchheim > Germany > email: sebastian.hu...@embedded-brains.de > phone: +49-89-18 94 741 - 16 > fax: +49-89-18 94 741 - 08 > > Registergericht: Amtsgericht München > Registernummer: HRB 157899 > Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler > Unsere Datenschutzerklärung finden Sie hier: > https://embedded-brains.de/datenschutzerklaerung/ > ___ > devel mailing list > devel@rtems.org > http://lists.rtems.org/mailman/listinfo/devel ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 2/5] build: Use CSafeLoader if available
On 27.04.23 02:11, Chris Johns wrote: On 26/4/2023 6:04 pm, Sebastian Huber wrote: The CSafeLoader uses the C libyaml libary to considerably speed up the loading of YAML files. No from me. What do you mean with not for me? You have the CSafeLoader available and it is slow? Do you have some timings before and after the patch set for a "./waf configure" and "./waf build"? On my systems the configure needs less than a second with the CSafeLoader and the waf build setup time is less than 100ms. I do not agree with conditional states of operation in the build system that depend on packages a host has installed. If speed is an important factor all users then I suggest you find a means to have it available automatically on the hosts we support (Linux, FreeBSD, MacOS, Windows MINGW64 and Cygwin. I am not sure if we should automatically install system Python packages on user machines. The fall back is the Python PyYAML package available through the RTEMS sources. This is what we use currently. For RTEMS users, this is acceptable since they are not supposed to touch the YAML files. For RTEMS maintainers, not having the cache makes working with the build system more efficient. If they system PyYAML package is not installed, then you get now a hint to install it: Setting top to : /home/EB/sebastian_h/src/rtems Setting out to : /home/EB/sebastian_h/src/rtems/build Regenerate the build specification cache. Install the PyYAML Python package to avoid this. The cache regeneration needs a couple of seconds... Configure board support package (BSP): arm/realview_pbx_a9_qemu -- embedded brains GmbH Herr Sebastian HUBER Dornierstr. 4 82178 Puchheim Germany email: sebastian.hu...@embedded-brains.de phone: +49-89-18 94 741 - 16 fax: +49-89-18 94 741 - 08 Registergericht: Amtsgericht München Registernummer: HRB 157899 Vertretungsberechtigte Geschäftsführer: Peter Rasmussen, Thomas Dörfler Unsere Datenschutzerklärung finden Sie hier: https://embedded-brains.de/datenschutzerklaerung/ ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
Re: [PATCH 2/5] build: Use CSafeLoader if available
On 26/4/2023 6:04 pm, Sebastian Huber wrote: > The CSafeLoader uses the C libyaml libary to considerably speed up the > loading of YAML files. No from me. I do not agree with conditional states of operation in the build system that depend on packages a host has installed. If speed is an important factor all users then I suggest you find a means to have it available automatically on the hosts we support (Linux, FreeBSD, MacOS, Windows MINGW64 and Cygwin. Chris ___ devel mailing list devel@rtems.org http://lists.rtems.org/mailman/listinfo/devel
