Re: General Infrastructure Maintainability: API and EBN
On Mon, Nov 11, 2019 at 12:41 AM Albert Astals Cid wrote: > > El diumenge, 10 de novembre de 2019, a les 1:48:10 CET, Ben Cooksley va > escriure: > > On Sun, Nov 10, 2019 at 11:20 AM Albert Astals Cid wrote: > > > > > > El dissabte, 9 de novembre de 2019, a les 22:03:37 CET, Ben Cooksley va > > > escriure: > > > > On Sun, Nov 10, 2019 at 9:29 AM Alexander Potashev > > > > wrote: > > > > > > > > > > сб, 9 нояб. 2019 г. в 21:50, Ben Cooksley : > > > > > > > > Over the past number of years one of the tasks the Sysadmin > > > > > > > > team has > > > > > > > > worked on has been improving the overall maintainability of our > > > > > > > > systems, with a significant number of specialised cronjobs, > > > > > > > > exceptions > > > > > > > > and hidden linkages being eliminated. > > > > > > > > > > > > > > > > That is with one great exception: api.kde.org and ebn.kde.org. > > > > > > > > > > > > > > > > Both of these are suffering from an extreme amount of digital > > > > > > > > bitrot > > > > > > > > and special casing and in general are now in a condition where I > > > > > > > > cannot say for certain whether it would be possible to > > > > > > > > replicate the > > > > > > > > setup on a new system without us experiencing some degree of > > > > > > > > breakage > > > > > > > > (some of which we may not discover until weeks/months > > > > > > > > afterwards). > > > > > > > > > > > > > > > > In addition, the current setup relies on an old-fashioned > > > > > > > > overnight > > > > > > > > reprocessing of all repositories, which is inefficient and > > > > > > > > resource > > > > > > > > expensive. A more modern approach would have the various > > > > > > > > projects api > > > > > > > > documentation generated on a delayed cycle from relevant > > > > > > > > branches as > > > > > > > > part of something like a CI job (but not part of the actual CI > > > > > > > > workflow itself). > > > > > > > > > > Hi Ben, > > > > > > > > > > I can't discuss this topic before we understand what exactly is wrong > > > > > with api.kde.org and ebn.kde.org and why they are hard to manage. > > > > > Could you please describe the current situation (where to find source > > > > > code, how many servers, etc) in new Phabricator tasks like you did for > > > > > identity.kde.org in https://phabricator.kde.org/T8449 ? > > > > > > > > > > P.S. Complicated legacy systems can be easy to replicate once you > > > > > automate their deployment by using Docker containers and/or > > > > > configuration management software like Ansible. > > > > > > > > The problem isn't just the complicated legacy nature of them, but also > > > > how fragile and impossible to maintain they are. > > > > > > > > There are currently to my knowledge the following ways of generating > > > > documentation within the system as it currently stands: > > > > - Legacy KDE 4 era generation tooling, still in use for large parts of > > > > KF5 based applications (which needs periodic fixes, see > > > > https://cgit.kde.org/kdelibs.git/commit/?id=12a8bb503351b98869f722ba932f822e3c495883) > > > > - DoxyQML, which is in part reliant on the above KDE 4 era generation > > > > tooling > > > > - Specialist handling for 'cmakedocs' and 'mkdocs' based projects > > > > - New era KApiDox tooling > > > > > > > > On top of all of this, the entire process can only be run as one > > > > single monolithic piece, which makes debugging and investigating > > > > faults difficult. > > > > > > > > To use an example of this, back in February 2018 we received a request > > > > (Sysadmin ticket) from someone reporting that their project's API > > > > documentation wasn't being generated. > > > > Despite investigation by Allen and others, we were unable to resolve > > > > the issue, and recommended that the project in question be switched > > > > from the KDE 4 era tooling to the newer KApiDox tooling, as it wasn't > > > > possible to identify the fault. > > > > > > > > Earlier this year there was a fault with API generation in general > > > > which broke a number of projects due to encoding of filenames/folders > > > > (fixed in > > > > https://cgit.kde.org/websites/quality-kde-org.git/commit/?id=f71c68bc80cce68aa3bcc6a51626c8f22b7c73c3). > > > > > > I volunteer to fix it. > > > > Hi Albert, > > > > Thanks for the offer of help. > > > > Please note though that this is much more than just fixing what we > > currently have. > > It is re-engineering what we currently have to ensure that it is > > maintainable, and meets the broader needs of the community in the long > > term. > > Yes, i understand that. > > > > > What we have currently doesn't really meet those standards - and as I > > mentioned earlier, is in some cases impossible to debug and therefore > > fix as everything appears to be working properly but ends up not > > working. > > > > It would also be nice if we could use this as an opportunity for > > someone not yet deeply involved to get involved more closely. > >
Re: General Infrastructure Maintainability: API and EBN
For EBN, personally I don't find myself using it. I think clang / llvm warnings / new Qt warnings cover a large part of the source code section. I've made a task listing everything EBN currently checks. Then together can mark off what things have other replacement, and what remaining things are actually still useful: https://phabricator.kde.org/T12013 David
Re: General Infrastructure Maintainability: API and EBN
Hi, Le dimanche 10 novembre 2019, 00:42:10 CET Philippe Cloutier a écrit : > On 2019-11-09 13:40, Ben Cooksley wrote: > > On Sun, Nov 10, 2019 at 4:28 AM Elvis Angelaccio > > wrote: > >> > >> > >> On 09/11/19 01:33, Ben Cooksley wrote: > >>> Hi all, > >> Hi Ben > > Hi Elvis, > > > >>> Over the past number of years one of the tasks the Sysadmin team has > >>> worked on has been improving the overall maintainability of our > >>> systems, with a significant number of specialised cronjobs, exceptions > >>> and hidden linkages being eliminated. > >>> > >>> That is with one great exception: api.kde.org and ebn.kde.org. > >>> > >>> Both of these are suffering from an extreme amount of digital bitrot > >>> and special casing and in general are now in a condition where I > >>> cannot say for certain whether it would be possible to replicate the > >>> setup on a new system without us experiencing some degree of breakage > >>> (some of which we may not discover until weeks/months afterwards). > >>> > >>> In addition, the current setup relies on an old-fashioned overnight > >>> reprocessing of all repositories, which is inefficient and resource > >>> expensive. A more modern approach would have the various projects api > >>> documentation generated on a delayed cycle from relevant branches as > >>> part of something like a CI job (but not part of the actual CI > >>> workflow itself). > >>> > >>> For this one, i'm not certain on the best path forward at this stage, > >>> however the current state of affairs cannot continue. We have tried > >>> over the past few years to find people to work on a replacement for > >>> the tooling involved, but alas we've yet to have success here. > >>> > >>> Thoughts anyone? > >> I'd say api.kde.org is too important to let it go. The EBN is less > >> important but still useful, I still see people pushing EBN-based fixes > >> once in a while. > >> > >> Have we ever tried to use a GSoC project to modernize either of them? If > >> not, would it make sense to try next year? > >> > > My only concern there would be whether the project is large enough in > > scope to justify the amount of time commitment a GSoC project normally > > spans. > > From my understanding of the problem in question it quite probably > > does not meet those requirements. > > Do you have an evaluation of the time necessary (and for a student)? > > I imagine there would be lots of work on api.kde.org possible if a > summer is too long for just that. Do we still have a list of api.kde.org > issues (it seems issues moved away from bugs.kde.org and I can no longer > find a list)? > The issues from kapidox are on bugs.kde.org as the product "frameworks-kapidox". There are not so many reported bugs. other parts of api.kde.org are not really tracked it seems. Olivier > > > > >>> Regards, > >>> Ben > >> Cheers, > >> Elvis > >> > > Cheers, > > Ben > >
Re: General Infrastructure Maintainability: API and EBN
El diumenge, 10 de novembre de 2019, a les 1:48:10 CET, Ben Cooksley va escriure: > On Sun, Nov 10, 2019 at 11:20 AM Albert Astals Cid wrote: > > > > El dissabte, 9 de novembre de 2019, a les 22:03:37 CET, Ben Cooksley va > > escriure: > > > On Sun, Nov 10, 2019 at 9:29 AM Alexander Potashev > > > wrote: > > > > > > > > сб, 9 нояб. 2019 г. в 21:50, Ben Cooksley : > > > > > > > Over the past number of years one of the tasks the Sysadmin team > > > > > > > has > > > > > > > worked on has been improving the overall maintainability of our > > > > > > > systems, with a significant number of specialised cronjobs, > > > > > > > exceptions > > > > > > > and hidden linkages being eliminated. > > > > > > > > > > > > > > That is with one great exception: api.kde.org and ebn.kde.org. > > > > > > > > > > > > > > Both of these are suffering from an extreme amount of digital > > > > > > > bitrot > > > > > > > and special casing and in general are now in a condition where I > > > > > > > cannot say for certain whether it would be possible to replicate > > > > > > > the > > > > > > > setup on a new system without us experiencing some degree of > > > > > > > breakage > > > > > > > (some of which we may not discover until weeks/months afterwards). > > > > > > > > > > > > > > In addition, the current setup relies on an old-fashioned > > > > > > > overnight > > > > > > > reprocessing of all repositories, which is inefficient and > > > > > > > resource > > > > > > > expensive. A more modern approach would have the various projects > > > > > > > api > > > > > > > documentation generated on a delayed cycle from relevant branches > > > > > > > as > > > > > > > part of something like a CI job (but not part of the actual CI > > > > > > > workflow itself). > > > > > > > > Hi Ben, > > > > > > > > I can't discuss this topic before we understand what exactly is wrong > > > > with api.kde.org and ebn.kde.org and why they are hard to manage. > > > > Could you please describe the current situation (where to find source > > > > code, how many servers, etc) in new Phabricator tasks like you did for > > > > identity.kde.org in https://phabricator.kde.org/T8449 ? > > > > > > > > P.S. Complicated legacy systems can be easy to replicate once you > > > > automate their deployment by using Docker containers and/or > > > > configuration management software like Ansible. > > > > > > The problem isn't just the complicated legacy nature of them, but also > > > how fragile and impossible to maintain they are. > > > > > > There are currently to my knowledge the following ways of generating > > > documentation within the system as it currently stands: > > > - Legacy KDE 4 era generation tooling, still in use for large parts of > > > KF5 based applications (which needs periodic fixes, see > > > https://cgit.kde.org/kdelibs.git/commit/?id=12a8bb503351b98869f722ba932f822e3c495883) > > > - DoxyQML, which is in part reliant on the above KDE 4 era generation > > > tooling > > > - Specialist handling for 'cmakedocs' and 'mkdocs' based projects > > > - New era KApiDox tooling > > > > > > On top of all of this, the entire process can only be run as one > > > single monolithic piece, which makes debugging and investigating > > > faults difficult. > > > > > > To use an example of this, back in February 2018 we received a request > > > (Sysadmin ticket) from someone reporting that their project's API > > > documentation wasn't being generated. > > > Despite investigation by Allen and others, we were unable to resolve > > > the issue, and recommended that the project in question be switched > > > from the KDE 4 era tooling to the newer KApiDox tooling, as it wasn't > > > possible to identify the fault. > > > > > > Earlier this year there was a fault with API generation in general > > > which broke a number of projects due to encoding of filenames/folders > > > (fixed in > > > https://cgit.kde.org/websites/quality-kde-org.git/commit/?id=f71c68bc80cce68aa3bcc6a51626c8f22b7c73c3). > > > > I volunteer to fix it. > > Hi Albert, > > Thanks for the offer of help. > > Please note though that this is much more than just fixing what we > currently have. > It is re-engineering what we currently have to ensure that it is > maintainable, and meets the broader needs of the community in the long > term. Yes, i understand that. > > What we have currently doesn't really meet those standards - and as I > mentioned earlier, is in some cases impossible to debug and therefore > fix as everything appears to be working properly but ends up not > working. > > It would also be nice if we could use this as an opportunity for > someone not yet deeply involved to get involved more closely. That's kind of impossible though, asking a total newbie to take care of re-engineering a "old system" is almost a recipe for 0% hands raised. Cheers, Albert > > > > > Cheers, > > Albert > > > > Cheers, > Ben > > > > > > > > > > > > -- > > > > Alexander Potashev > > > > > >
Re: General Infrastructure Maintainability: API and EBN
On Sun, Nov 10, 2019 at 10:22 AM Adriaan de Groot wrote: > > On Saturday, November 9, 2019 10:03:37 PM CET Ben Cooksley wrote: > > The problem isn't just the complicated legacy nature of them, but also > > how fragile and impossible to maintain they are. > > Just as an example, the EBN (hm .. I no longer redirect EBN.org to the KDE > site) still embeds a comment in the front page about a research project that > ended in 2006 and points to my blog that stopped working in 2011. > > I like Alexander's suggestion to make a Phab task for it. I'd dearly love to > spend some time reinvigorating this stuff, but .. time is precious. I remember > when re-tooling APIDOX around 2010 that it worked fairly effectively *because* > we (the university of Nijmegen at the time) had a full SVN repo mirror locally > for research purposes. That's no longer the case. I've now written up what will be needed for this at https://phabricator.kde.org/T12004 Fortunately time wise I don't think it will be too intensive. Assuming no major issues are hit with Part 1 (the actual generator) that is - although given it currently works for the most part with Frameworks already, it should be fine with about the only complicated part potentially being QCH support. Part 3 will be a case of adapting stuff we already have (i'd estimate about 2 hours worth of work, maximum) and Part 2 once a HTML template is available should be a case of scanning YAML format files and then rendering out the necessary information. > > [ade] Cheers, Ben
Re: General Infrastructure Maintainability: API and EBN
On Sun, Nov 10, 2019 at 12:42 PM Philippe Cloutier wrote: > > On 2019-11-09 13:40, Ben Cooksley wrote: > > On Sun, Nov 10, 2019 at 4:28 AM Elvis Angelaccio > > wrote: > >> > >> > >> On 09/11/19 01:33, Ben Cooksley wrote: > >>> Hi all, > >> Hi Ben > > Hi Elvis, > > > >>> Over the past number of years one of the tasks the Sysadmin team has > >>> worked on has been improving the overall maintainability of our > >>> systems, with a significant number of specialised cronjobs, exceptions > >>> and hidden linkages being eliminated. > >>> > >>> That is with one great exception: api.kde.org and ebn.kde.org. > >>> > >>> Both of these are suffering from an extreme amount of digital bitrot > >>> and special casing and in general are now in a condition where I > >>> cannot say for certain whether it would be possible to replicate the > >>> setup on a new system without us experiencing some degree of breakage > >>> (some of which we may not discover until weeks/months afterwards). > >>> > >>> In addition, the current setup relies on an old-fashioned overnight > >>> reprocessing of all repositories, which is inefficient and resource > >>> expensive. A more modern approach would have the various projects api > >>> documentation generated on a delayed cycle from relevant branches as > >>> part of something like a CI job (but not part of the actual CI > >>> workflow itself). > >>> > >>> For this one, i'm not certain on the best path forward at this stage, > >>> however the current state of affairs cannot continue. We have tried > >>> over the past few years to find people to work on a replacement for > >>> the tooling involved, but alas we've yet to have success here. > >>> > >>> Thoughts anyone? > >> I'd say api.kde.org is too important to let it go. The EBN is less > >> important but still useful, I still see people pushing EBN-based fixes > >> once in a while. > >> > >> Have we ever tried to use a GSoC project to modernize either of them? If > >> not, would it make sense to try next year? > >> > > My only concern there would be whether the project is large enough in > > scope to justify the amount of time commitment a GSoC project normally > > spans. > > From my understanding of the problem in question it quite probably > > does not meet those requirements. > > Do you have an evaluation of the time necessary (and for a student)? That is a little hard to estimate, but I will see if it can be done as part of a write up that Alexander and Adriaan asked for. > > I imagine there would be lots of work on api.kde.org possible if a > summer is too long for just that. Do we still have a list of api.kde.org > issues (it seems issues moved away from bugs.kde.org and I can no longer > find a list)? > > Regards, Ben > > > >>> Regards, > >>> Ben > >> Cheers, > >> Elvis > >> > > Cheers, > > Ben > > -- > Philippe Cloutier > http://www.philippecloutier.com >
Re: General Infrastructure Maintainability: API and EBN
On Sun, Nov 10, 2019 at 11:20 AM Albert Astals Cid wrote: > > El dissabte, 9 de novembre de 2019, a les 22:03:37 CET, Ben Cooksley va > escriure: > > On Sun, Nov 10, 2019 at 9:29 AM Alexander Potashev > > wrote: > > > > > > сб, 9 нояб. 2019 г. в 21:50, Ben Cooksley : > > > > > > Over the past number of years one of the tasks the Sysadmin team has > > > > > > worked on has been improving the overall maintainability of our > > > > > > systems, with a significant number of specialised cronjobs, > > > > > > exceptions > > > > > > and hidden linkages being eliminated. > > > > > > > > > > > > That is with one great exception: api.kde.org and ebn.kde.org. > > > > > > > > > > > > Both of these are suffering from an extreme amount of digital bitrot > > > > > > and special casing and in general are now in a condition where I > > > > > > cannot say for certain whether it would be possible to replicate the > > > > > > setup on a new system without us experiencing some degree of > > > > > > breakage > > > > > > (some of which we may not discover until weeks/months afterwards). > > > > > > > > > > > > In addition, the current setup relies on an old-fashioned overnight > > > > > > reprocessing of all repositories, which is inefficient and resource > > > > > > expensive. A more modern approach would have the various projects > > > > > > api > > > > > > documentation generated on a delayed cycle from relevant branches as > > > > > > part of something like a CI job (but not part of the actual CI > > > > > > workflow itself). > > > > > > Hi Ben, > > > > > > I can't discuss this topic before we understand what exactly is wrong > > > with api.kde.org and ebn.kde.org and why they are hard to manage. > > > Could you please describe the current situation (where to find source > > > code, how many servers, etc) in new Phabricator tasks like you did for > > > identity.kde.org in https://phabricator.kde.org/T8449 ? > > > > > > P.S. Complicated legacy systems can be easy to replicate once you > > > automate their deployment by using Docker containers and/or > > > configuration management software like Ansible. > > > > The problem isn't just the complicated legacy nature of them, but also > > how fragile and impossible to maintain they are. > > > > There are currently to my knowledge the following ways of generating > > documentation within the system as it currently stands: > > - Legacy KDE 4 era generation tooling, still in use for large parts of > > KF5 based applications (which needs periodic fixes, see > > https://cgit.kde.org/kdelibs.git/commit/?id=12a8bb503351b98869f722ba932f822e3c495883) > > - DoxyQML, which is in part reliant on the above KDE 4 era generation > > tooling > > - Specialist handling for 'cmakedocs' and 'mkdocs' based projects > > - New era KApiDox tooling > > > > On top of all of this, the entire process can only be run as one > > single monolithic piece, which makes debugging and investigating > > faults difficult. > > > > To use an example of this, back in February 2018 we received a request > > (Sysadmin ticket) from someone reporting that their project's API > > documentation wasn't being generated. > > Despite investigation by Allen and others, we were unable to resolve > > the issue, and recommended that the project in question be switched > > from the KDE 4 era tooling to the newer KApiDox tooling, as it wasn't > > possible to identify the fault. > > > > Earlier this year there was a fault with API generation in general > > which broke a number of projects due to encoding of filenames/folders > > (fixed in > > https://cgit.kde.org/websites/quality-kde-org.git/commit/?id=f71c68bc80cce68aa3bcc6a51626c8f22b7c73c3). > > I volunteer to fix it. Hi Albert, Thanks for the offer of help. Please note though that this is much more than just fixing what we currently have. It is re-engineering what we currently have to ensure that it is maintainable, and meets the broader needs of the community in the long term. What we have currently doesn't really meet those standards - and as I mentioned earlier, is in some cases impossible to debug and therefore fix as everything appears to be working properly but ends up not working. It would also be nice if we could use this as an opportunity for someone not yet deeply involved to get involved more closely. > > Cheers, > Albert > Cheers, Ben > > > > > > > > -- > > > Alexander Potashev > > > > Cheers, > > Ben > > > > > >
Re: General Infrastructure Maintainability: API and EBN
On 2019-11-09 13:40, Ben Cooksley wrote: On Sun, Nov 10, 2019 at 4:28 AM Elvis Angelaccio wrote: On 09/11/19 01:33, Ben Cooksley wrote: Hi all, Hi Ben Hi Elvis, Over the past number of years one of the tasks the Sysadmin team has worked on has been improving the overall maintainability of our systems, with a significant number of specialised cronjobs, exceptions and hidden linkages being eliminated. That is with one great exception: api.kde.org and ebn.kde.org. Both of these are suffering from an extreme amount of digital bitrot and special casing and in general are now in a condition where I cannot say for certain whether it would be possible to replicate the setup on a new system without us experiencing some degree of breakage (some of which we may not discover until weeks/months afterwards). In addition, the current setup relies on an old-fashioned overnight reprocessing of all repositories, which is inefficient and resource expensive. A more modern approach would have the various projects api documentation generated on a delayed cycle from relevant branches as part of something like a CI job (but not part of the actual CI workflow itself). For this one, i'm not certain on the best path forward at this stage, however the current state of affairs cannot continue. We have tried over the past few years to find people to work on a replacement for the tooling involved, but alas we've yet to have success here. Thoughts anyone? I'd say api.kde.org is too important to let it go. The EBN is less important but still useful, I still see people pushing EBN-based fixes once in a while. Have we ever tried to use a GSoC project to modernize either of them? If not, would it make sense to try next year? My only concern there would be whether the project is large enough in scope to justify the amount of time commitment a GSoC project normally spans. From my understanding of the problem in question it quite probably does not meet those requirements. Do you have an evaluation of the time necessary (and for a student)? I imagine there would be lots of work on api.kde.org possible if a summer is too long for just that. Do we still have a list of api.kde.org issues (it seems issues moved away from bugs.kde.org and I can no longer find a list)? Regards, Ben Cheers, Elvis Cheers, Ben -- Philippe Cloutier http://www.philippecloutier.com
Re: General Infrastructure Maintainability: API and EBN
El dissabte, 9 de novembre de 2019, a les 22:03:37 CET, Ben Cooksley va escriure: > On Sun, Nov 10, 2019 at 9:29 AM Alexander Potashev > wrote: > > > > сб, 9 нояб. 2019 г. в 21:50, Ben Cooksley : > > > > > Over the past number of years one of the tasks the Sysadmin team has > > > > > worked on has been improving the overall maintainability of our > > > > > systems, with a significant number of specialised cronjobs, exceptions > > > > > and hidden linkages being eliminated. > > > > > > > > > > That is with one great exception: api.kde.org and ebn.kde.org. > > > > > > > > > > Both of these are suffering from an extreme amount of digital bitrot > > > > > and special casing and in general are now in a condition where I > > > > > cannot say for certain whether it would be possible to replicate the > > > > > setup on a new system without us experiencing some degree of breakage > > > > > (some of which we may not discover until weeks/months afterwards). > > > > > > > > > > In addition, the current setup relies on an old-fashioned overnight > > > > > reprocessing of all repositories, which is inefficient and resource > > > > > expensive. A more modern approach would have the various projects api > > > > > documentation generated on a delayed cycle from relevant branches as > > > > > part of something like a CI job (but not part of the actual CI > > > > > workflow itself). > > > > Hi Ben, > > > > I can't discuss this topic before we understand what exactly is wrong > > with api.kde.org and ebn.kde.org and why they are hard to manage. > > Could you please describe the current situation (where to find source > > code, how many servers, etc) in new Phabricator tasks like you did for > > identity.kde.org in https://phabricator.kde.org/T8449 ? > > > > P.S. Complicated legacy systems can be easy to replicate once you > > automate their deployment by using Docker containers and/or > > configuration management software like Ansible. > > The problem isn't just the complicated legacy nature of them, but also > how fragile and impossible to maintain they are. > > There are currently to my knowledge the following ways of generating > documentation within the system as it currently stands: > - Legacy KDE 4 era generation tooling, still in use for large parts of > KF5 based applications (which needs periodic fixes, see > https://cgit.kde.org/kdelibs.git/commit/?id=12a8bb503351b98869f722ba932f822e3c495883) > - DoxyQML, which is in part reliant on the above KDE 4 era generation tooling > - Specialist handling for 'cmakedocs' and 'mkdocs' based projects > - New era KApiDox tooling > > On top of all of this, the entire process can only be run as one > single monolithic piece, which makes debugging and investigating > faults difficult. > > To use an example of this, back in February 2018 we received a request > (Sysadmin ticket) from someone reporting that their project's API > documentation wasn't being generated. > Despite investigation by Allen and others, we were unable to resolve > the issue, and recommended that the project in question be switched > from the KDE 4 era tooling to the newer KApiDox tooling, as it wasn't > possible to identify the fault. > > Earlier this year there was a fault with API generation in general > which broke a number of projects due to encoding of filenames/folders > (fixed in > https://cgit.kde.org/websites/quality-kde-org.git/commit/?id=f71c68bc80cce68aa3bcc6a51626c8f22b7c73c3). I volunteer to fix it. Cheers, Albert > > > > > -- > > Alexander Potashev > > Cheers, > Ben >
Re: General Infrastructure Maintainability: API and EBN
On Saturday, November 9, 2019 10:03:37 PM CET Ben Cooksley wrote: > The problem isn't just the complicated legacy nature of them, but also > how fragile and impossible to maintain they are. Just as an example, the EBN (hm .. I no longer redirect EBN.org to the KDE site) still embeds a comment in the front page about a research project that ended in 2006 and points to my blog that stopped working in 2011. I like Alexander's suggestion to make a Phab task for it. I'd dearly love to spend some time reinvigorating this stuff, but .. time is precious. I remember when re-tooling APIDOX around 2010 that it worked fairly effectively *because* we (the university of Nijmegen at the time) had a full SVN repo mirror locally for research purposes. That's no longer the case. [ade] signature.asc Description: This is a digitally signed message part.
Re: General Infrastructure Maintainability: API and EBN
On Sun, Nov 10, 2019 at 9:29 AM Alexander Potashev wrote: > > сб, 9 нояб. 2019 г. в 21:50, Ben Cooksley : > > > > Over the past number of years one of the tasks the Sysadmin team has > > > > worked on has been improving the overall maintainability of our > > > > systems, with a significant number of specialised cronjobs, exceptions > > > > and hidden linkages being eliminated. > > > > > > > > That is with one great exception: api.kde.org and ebn.kde.org. > > > > > > > > Both of these are suffering from an extreme amount of digital bitrot > > > > and special casing and in general are now in a condition where I > > > > cannot say for certain whether it would be possible to replicate the > > > > setup on a new system without us experiencing some degree of breakage > > > > (some of which we may not discover until weeks/months afterwards). > > > > > > > > In addition, the current setup relies on an old-fashioned overnight > > > > reprocessing of all repositories, which is inefficient and resource > > > > expensive. A more modern approach would have the various projects api > > > > documentation generated on a delayed cycle from relevant branches as > > > > part of something like a CI job (but not part of the actual CI > > > > workflow itself). > > Hi Ben, > > I can't discuss this topic before we understand what exactly is wrong > with api.kde.org and ebn.kde.org and why they are hard to manage. > Could you please describe the current situation (where to find source > code, how many servers, etc) in new Phabricator tasks like you did for > identity.kde.org in https://phabricator.kde.org/T8449 ? > > P.S. Complicated legacy systems can be easy to replicate once you > automate their deployment by using Docker containers and/or > configuration management software like Ansible. The problem isn't just the complicated legacy nature of them, but also how fragile and impossible to maintain they are. There are currently to my knowledge the following ways of generating documentation within the system as it currently stands: - Legacy KDE 4 era generation tooling, still in use for large parts of KF5 based applications (which needs periodic fixes, see https://cgit.kde.org/kdelibs.git/commit/?id=12a8bb503351b98869f722ba932f822e3c495883) - DoxyQML, which is in part reliant on the above KDE 4 era generation tooling - Specialist handling for 'cmakedocs' and 'mkdocs' based projects - New era KApiDox tooling On top of all of this, the entire process can only be run as one single monolithic piece, which makes debugging and investigating faults difficult. To use an example of this, back in February 2018 we received a request (Sysadmin ticket) from someone reporting that their project's API documentation wasn't being generated. Despite investigation by Allen and others, we were unable to resolve the issue, and recommended that the project in question be switched from the KDE 4 era tooling to the newer KApiDox tooling, as it wasn't possible to identify the fault. Earlier this year there was a fault with API generation in general which broke a number of projects due to encoding of filenames/folders (fixed in https://cgit.kde.org/websites/quality-kde-org.git/commit/?id=f71c68bc80cce68aa3bcc6a51626c8f22b7c73c3). > > -- > Alexander Potashev Cheers, Ben
Re: General Infrastructure Maintainability: API and EBN
сб, 9 нояб. 2019 г. в 21:50, Ben Cooksley : > > > Over the past number of years one of the tasks the Sysadmin team has > > > worked on has been improving the overall maintainability of our > > > systems, with a significant number of specialised cronjobs, exceptions > > > and hidden linkages being eliminated. > > > > > > That is with one great exception: api.kde.org and ebn.kde.org. > > > > > > Both of these are suffering from an extreme amount of digital bitrot > > > and special casing and in general are now in a condition where I > > > cannot say for certain whether it would be possible to replicate the > > > setup on a new system without us experiencing some degree of breakage > > > (some of which we may not discover until weeks/months afterwards). > > > > > > In addition, the current setup relies on an old-fashioned overnight > > > reprocessing of all repositories, which is inefficient and resource > > > expensive. A more modern approach would have the various projects api > > > documentation generated on a delayed cycle from relevant branches as > > > part of something like a CI job (but not part of the actual CI > > > workflow itself). Hi Ben, I can't discuss this topic before we understand what exactly is wrong with api.kde.org and ebn.kde.org and why they are hard to manage. Could you please describe the current situation (where to find source code, how many servers, etc) in new Phabricator tasks like you did for identity.kde.org in https://phabricator.kde.org/T8449 ? P.S. Complicated legacy systems can be easy to replicate once you automate their deployment by using Docker containers and/or configuration management software like Ansible. -- Alexander Potashev
Re: General Infrastructure Maintainability: API and EBN
On Sun, Nov 10, 2019 at 7:22 AM Philippe Cloutier wrote: > > Hi Ben, Hi Philippe, > > On 2019-11-08 19:33, Ben Cooksley wrote: > > Hi all, > > > > Over the past number of years one of the tasks the Sysadmin team has > > worked on has been improving the overall maintainability of our > > systems, with a significant number of specialised cronjobs, exceptions > > and hidden linkages being eliminated. > > > > That is with one great exception: api.kde.org and ebn.kde.org. > > > > Both of these are suffering from an extreme amount of digital bitrot > > and special casing and in general are now in a condition where I > > cannot say for certain whether it would be possible to replicate the > > setup on a new system without us experiencing some degree of breakage > > (some of which we may not discover until weeks/months afterwards). > > > > In addition, the current setup relies on an old-fashioned overnight > > reprocessing of all repositories, which is inefficient and resource > > expensive. A more modern approach would have the various projects api > > documentation generated on a delayed cycle from relevant branches as > > part of something like a CI job (but not part of the actual CI > > workflow itself). > > > > For this one, i'm not certain on the best path forward at this stage, > > however the current state of affairs cannot continue. We have tried > > over the past few years to find people to work on a replacement for > > the tooling involved, but alas we've yet to have success here. > > > > Thoughts anyone? > > Thank you for bringing up this topic. I last used api.kde.org a few > months ago (a few tens of hits) and I appreciated it. > > I'm pretty sure we should keep offering a solution to the problem it > addresses, but usage statistics would be good. Thankfully, I believe the > service api.kde.org provides is not something its users will develop > much dependency towards. If a catastrophe happened, I don't see a big > issue in stopping to provide the service for the time it takes to > restore it (beyond the issue of simply not providing the service during > that time). > > Regarding resource usage, has it been quantified somewhat? We are aware that it is very much in use, but no attempt has been made to gather statistics (in part because embedding the necessary stats.kde.org tags to do website statistics collection is not straight forward with the existing setup). This thread is to discuss how we might fix the problem with maintaining the generation process. > > Perhaps I can suggest simply adding some banner to the site or at least > its homepage indicating that help is needed, with a link to a page > discussing the attempts you mention. > > > > > Regards, > > Ben > > -- > Philippe Cloutier > http://www.philippecloutier.com > Regards, Ben
Re: General Infrastructure Maintainability: API and EBN
On Sun, Nov 10, 2019 at 4:28 AM Elvis Angelaccio wrote: > > > > On 09/11/19 01:33, Ben Cooksley wrote: > > Hi all, > > Hi Ben Hi Elvis, > > > > > Over the past number of years one of the tasks the Sysadmin team has > > worked on has been improving the overall maintainability of our > > systems, with a significant number of specialised cronjobs, exceptions > > and hidden linkages being eliminated. > > > > That is with one great exception: api.kde.org and ebn.kde.org. > > > > Both of these are suffering from an extreme amount of digital bitrot > > and special casing and in general are now in a condition where I > > cannot say for certain whether it would be possible to replicate the > > setup on a new system without us experiencing some degree of breakage > > (some of which we may not discover until weeks/months afterwards). > > > > In addition, the current setup relies on an old-fashioned overnight > > reprocessing of all repositories, which is inefficient and resource > > expensive. A more modern approach would have the various projects api > > documentation generated on a delayed cycle from relevant branches as > > part of something like a CI job (but not part of the actual CI > > workflow itself). > > > > For this one, i'm not certain on the best path forward at this stage, > > however the current state of affairs cannot continue. We have tried > > over the past few years to find people to work on a replacement for > > the tooling involved, but alas we've yet to have success here. > > > > Thoughts anyone? > > I'd say api.kde.org is too important to let it go. The EBN is less > important but still useful, I still see people pushing EBN-based fixes > once in a while. > > Have we ever tried to use a GSoC project to modernize either of them? If > not, would it make sense to try next year? > My only concern there would be whether the project is large enough in scope to justify the amount of time commitment a GSoC project normally spans. >From my understanding of the problem in question it quite probably does not meet those requirements. > > > > Regards, > > Ben > > Cheers, > Elvis > Cheers, Ben
Re: General Infrastructure Maintainability: API and EBN
Hi Ben, On 2019-11-08 19:33, Ben Cooksley wrote: Hi all, Over the past number of years one of the tasks the Sysadmin team has worked on has been improving the overall maintainability of our systems, with a significant number of specialised cronjobs, exceptions and hidden linkages being eliminated. That is with one great exception: api.kde.org and ebn.kde.org. Both of these are suffering from an extreme amount of digital bitrot and special casing and in general are now in a condition where I cannot say for certain whether it would be possible to replicate the setup on a new system without us experiencing some degree of breakage (some of which we may not discover until weeks/months afterwards). In addition, the current setup relies on an old-fashioned overnight reprocessing of all repositories, which is inefficient and resource expensive. A more modern approach would have the various projects api documentation generated on a delayed cycle from relevant branches as part of something like a CI job (but not part of the actual CI workflow itself). For this one, i'm not certain on the best path forward at this stage, however the current state of affairs cannot continue. We have tried over the past few years to find people to work on a replacement for the tooling involved, but alas we've yet to have success here. Thoughts anyone? Thank you for bringing up this topic. I last used api.kde.org a few months ago (a few tens of hits) and I appreciated it. I'm pretty sure we should keep offering a solution to the problem it addresses, but usage statistics would be good. Thankfully, I believe the service api.kde.org provides is not something its users will develop much dependency towards. If a catastrophe happened, I don't see a big issue in stopping to provide the service for the time it takes to restore it (beyond the issue of simply not providing the service during that time). Regarding resource usage, has it been quantified somewhat? Perhaps I can suggest simply adding some banner to the site or at least its homepage indicating that help is needed, with a link to a page discussing the attempts you mention. Regards, Ben -- Philippe Cloutier http://www.philippecloutier.com
Re: General Infrastructure Maintainability: API and EBN
Some of what ebn is doing in terms of linting c/c++ code would be better moved to clazy/clang-tidy plugins. The current scan for functions such open false positives for strings and comments. This is just open example of the limits imposed by not having a compilers eye view of the code. I personally don't care for having an extra place to check for errors. Clazy can run automatically on each build so warnings show up in your ide not just on some website. On Sat, Nov 9, 2019, 10:28 AM Elvis Angelaccio wrote: > > > On 09/11/19 01:33, Ben Cooksley wrote: > > Hi all, > > Hi Ben > > > > > Over the past number of years one of the tasks the Sysadmin team has > > worked on has been improving the overall maintainability of our > > systems, with a significant number of specialised cronjobs, exceptions > > and hidden linkages being eliminated. > > > > That is with one great exception: api.kde.org and ebn.kde.org. > > > > Both of these are suffering from an extreme amount of digital bitrot > > and special casing and in general are now in a condition where I > > cannot say for certain whether it would be possible to replicate the > > setup on a new system without us experiencing some degree of breakage > > (some of which we may not discover until weeks/months afterwards). > > > > In addition, the current setup relies on an old-fashioned overnight > > reprocessing of all repositories, which is inefficient and resource > > expensive. A more modern approach would have the various projects api > > documentation generated on a delayed cycle from relevant branches as > > part of something like a CI job (but not part of the actual CI > > workflow itself). > > > > For this one, i'm not certain on the best path forward at this stage, > > however the current state of affairs cannot continue. We have tried > > over the past few years to find people to work on a replacement for > > the tooling involved, but alas we've yet to have success here. > > > > Thoughts anyone? > > I'd say api.kde.org is too important to let it go. The EBN is less > important but still useful, I still see people pushing EBN-based fixes > once in a while. > > Have we ever tried to use a GSoC project to modernize either of them? If > not, would it make sense to try next year? > > > > > Regards, > > Ben > > Cheers, > Elvis > >
Re: General Infrastructure Maintainability: API and EBN
On 09/11/19 01:33, Ben Cooksley wrote: > Hi all, Hi Ben > > Over the past number of years one of the tasks the Sysadmin team has > worked on has been improving the overall maintainability of our > systems, with a significant number of specialised cronjobs, exceptions > and hidden linkages being eliminated. > > That is with one great exception: api.kde.org and ebn.kde.org. > > Both of these are suffering from an extreme amount of digital bitrot > and special casing and in general are now in a condition where I > cannot say for certain whether it would be possible to replicate the > setup on a new system without us experiencing some degree of breakage > (some of which we may not discover until weeks/months afterwards). > > In addition, the current setup relies on an old-fashioned overnight > reprocessing of all repositories, which is inefficient and resource > expensive. A more modern approach would have the various projects api > documentation generated on a delayed cycle from relevant branches as > part of something like a CI job (but not part of the actual CI > workflow itself). > > For this one, i'm not certain on the best path forward at this stage, > however the current state of affairs cannot continue. We have tried > over the past few years to find people to work on a replacement for > the tooling involved, but alas we've yet to have success here. > > Thoughts anyone? I'd say api.kde.org is too important to let it go. The EBN is less important but still useful, I still see people pushing EBN-based fixes once in a while. Have we ever tried to use a GSoC project to modernize either of them? If not, would it make sense to try next year? > > Regards, > Ben Cheers, Elvis
General Infrastructure Maintainability: API and EBN
Hi all, Over the past number of years one of the tasks the Sysadmin team has worked on has been improving the overall maintainability of our systems, with a significant number of specialised cronjobs, exceptions and hidden linkages being eliminated. That is with one great exception: api.kde.org and ebn.kde.org. Both of these are suffering from an extreme amount of digital bitrot and special casing and in general are now in a condition where I cannot say for certain whether it would be possible to replicate the setup on a new system without us experiencing some degree of breakage (some of which we may not discover until weeks/months afterwards). In addition, the current setup relies on an old-fashioned overnight reprocessing of all repositories, which is inefficient and resource expensive. A more modern approach would have the various projects api documentation generated on a delayed cycle from relevant branches as part of something like a CI job (but not part of the actual CI workflow itself). For this one, i'm not certain on the best path forward at this stage, however the current state of affairs cannot continue. We have tried over the past few years to find people to work on a replacement for the tooling involved, but alas we've yet to have success here. Thoughts anyone? Regards, Ben
