Re: [Kicad-developers] GitHub Plugin (my nemesis)

[David Godfrey]

On 29/09/17 01:53, Bernhard Stegmaier wrote: Just my 2 cents… I have seen some projects using submodules (open source and in my company). I have seen lots of problems and complaints (… doesn’t compile …) just because normal users or even normal developers (being no git experts) didn’t sync all submodules correctly. Thinking of how many problems are already seen now by users which are not used to version control systems (because they just don’t understand what’s happening under the hood) I wouldn’t want to force them to use git submodules… at least not as long as this is not completely hidden by some users friendly and dead simple KiCad GUI. Bernhard hit the nail on the head here. For normal Users, ALL of the git functionality should be hidden behind basic KiCad GUI features. However, for Users and Librarians that want to manage, add, edit at least a basic knowledge of whatever tooling is used behind the scenes is a HARD REQUIREMENT. These days git is probably one of the best documented, and most well supported in the greater community. That alone makes it a very good choice of backend. Handling of submodules can be slightly tricky, but a few simple helper scripts (for LedgerSMB project we use a Makefile with a few targets such as "submodules" which updates all submodules to the current repo head's commit references to them) Regards, Bernhard On 28. Sep 2017, at 13:29, Oliver Walters <oliver.henry.walt...@gmail.com> wrote: Simon, David, Thanks for the synopsis of your idea David, there are certainly some advantages to using the submodule approach. Is it github compatible?   Simon, yes git-submodule is compatible with github. I have a test repository here - https://github.com/kicad/kicad-footprints - this contains a submodule for each of the current footprint directories. Does it simplify the maintenance of the kicad library by the librarians? Compared to having all the .pretty folders as subdirectories of a single repo, no. Reorganising libraries or adding new ones is tedious, and for each of the 100+ .pretty repos, we (the librarians) have to manage: * Pull requests and isues * CI tools (e.g. Travis KLC scripts) * Updating each library when new requirements are enacted (e.g. KLC) * Ensure that changes in one repo do not conflict with a different repo * Handle reclassification or moving of footprints between repos It is also quite possible for submodules to become out of sync. The link that Simon posted has some a good example of how this can occur (detached head). I have some projects at work that contain about 5 submodules and it rapidly becomes difficult to deal with even for a competent user of Git. It becomes harder for users, too. KiCad users cannot, in the general case, be classified as competent users of git. Currently, users who wish to contribute to the footprint libraries have a hard time. For each library they want to contribute to, they have to fork and manage another repository. Personally I have done this for 50 separate repositories.  Further, once changes are made to each submodule, the master repository then needs to be updated otherwise it's not up to date with the latest changes. Perhaps we can use git hooks here, but this is just further complication! Let's also consider consistency. The footprint libraries are (currently) split into separate repos. However the symbol libraries and 3d model libraries are not. Should we split them too? Now we have 300+ repositories to manage. Let's not make this an academic problem and say "but it's possible to manage with this complex set of tools"! From the perspective of a contributor, and as the person who has to manage the libraries, the simpler the better.  Oliver On Thu, Sep 28, 2017 at 7:25 PM, Simon Küppers <simon.kuepp...@rub.de> wrote: Thanks for your detailed description. I think it is a nice way to go. However two remarks: Does it simplify the maintenance of the kicad library by the librarians? Is it github compatible? If not, we would need to find another platform to host the libraries. If I look for git submodule, there is a lot of different opinions on why you should or should not use it. (e.g. https://www.google.de/amp/s/codingkilledthecat.wordpress.com/2012/04/28/why-your-company-shouldnt-use-git-submodules/amp/). Does it apply in our case? What about git subtree? Best regards Simon Am 28. September 2017 02:37:09 MESZ schrieb David Godfrey <i...@sbts.com.au>: Hi All I have to agree with some of the other posters, why not use git as it was intended? Specifically I think the following makes sense..... - Keep the multiple repositories, this helps when a company only wants to make certain sets of libs available to it's staff - Have a master repository that includes all other repositories as submodules. - Have branches that are matched to kicad versions. This allows footprint changes in a version safe way. - Use standard "git clone" (initial download) and "git pull" (update) on the main repo which provides the entire set of available libs without actually downloading the content for all libs. - When a specific lib is needed, do similar to what we do now, and use "git submodule update --init --recursive $submoduleName" to just pull that specific submodule - Allow the "main" library repo URI to be altered. This enables a companies fork of the repo to be used. - Allow the "main" library repo URI to be ANY valid git URI. That means a repo on a local fileserver rather than a http server can be used. Along with various security options etc. - Add a fairly simple scripted tool that's run "on release" to retrieve a README.md (an possibly a descriptor/index file) from each actual library repo and update those within the "main" repo. This removes the need to have the current case where manual edits are required to keep the "main" repo in sync with what's available in the individual lib's. To add a new lib, it's then as simple as (from the main repo) doing something like.... - "git submodule add $URI" - "./scripts/update-library-indexes.sh" - "git commit -a -m $'added new module "modulename"\nupdated all module descriptions and index'" - "git push" All of the above allows the Main repo to be forked by a company (or individual) and have their own custom repo's added very easily. It even allows libraries to be easily excluded or replaced, all using extremely well developed management tools. On a side note, git submodules are stored in the main repository basically as URI that includes a commit reference. That means it's easy to specify a specific library version to include in a specific branch/tag of the main repo. As for the person that said "git isn't available for windows", sorry, that's bunkum. There are many many development environments out there that ONLY run on windows that have git integration. A quick google search for "git for windows" will show what you need to know there. Finally, wrapping git to duplicate the effective API that's currently in use should be relatively trivial, resulting in virtually no code changes required to KICAD it's self. Assuming the existing functionality is cleanly wrapped by an API, the only real change to KICAD would be the swapout of the module, and addition of an easy way to change the URI I know I haven't covered everything in this email, but it should be a good outline for further discussion On 22/09/17 09:21, Oliver Walters wrote: Hi all, Ok, now that the website integration with the libraries is (pretty much) done, and the licensing issue seems to be sorted, there is one final puzzle piece to solve before I'm happy with the state of the libraries for a v5 release. *Goal: *Merge all footprint library repositories into a single repo to solve the ongoing dramas of maintaining 100+ repos. *Problem*: The *only* thing standing in the way of just doing this is that some users like the GitHub plugin and previous instruction is that this functionality cannot be removed. The GitHub plugin functions by downloading a .zip file of a .pretty repo. If we merge all footprint libs into a single repo with multiple subdirectories, this will not work anymore (as GitHub dosen't allow you to download a .zip of a single subdirectory). Merging the repos is the *right thing to do*. But how to proceed? *Options:* * * /a) Drop github plugin feature, replace with library-download tool/ I don't think it is a good idea to live-load library data from GitHub (a lot of other users agree too). It's slow, and a waste of bandwidth to re-download the libs all the time. We drop support for loading libraries direct from GitHub. However, we add a tool for downloading libraries from GitHub and storing to disk. Users can update as they like. This can be integrated in KiCad and new users can run this tool when they first install KICad. This means that no libs need to distributed with the installer and users can update to latest libs whenever they want. /b) Improve github plugin to allow subdirectory traversal/ This is difficult and will only result in the plugin being slower. There are two ways I can see to do this: i. Use Git API - tools exist that use this functionality - https://github.com/KinoLien/gitzip ii. Use subversion - GitHub actually provides subversion API - https://www.seanw.org/blog/download-git-repo-subdirectory/ *Subversion* In either case, I think that using the subversion tool to partially download the libraries would be a good approach (I assume quicker than using wget and the GitHub API). 1. Does anyone have any experience using the C API for SVN? 2. The Python bindings are pretty good - https://pypi.python.org/pypi/svn - and much easier to use. However, can we make the library download tools dependent on enabling the python plugin? 3. Is there a way to checkout a subversion remote to memory (to replicate the functionality of the current GitHub plugin)? If not, I'm not sure how to approach option b) above. Feedback appreciated. I think that it is very important especially for new users that this is improved. The GitHub plugin is constantly causing headaches! Cheers, Oliver Regards David Godfrey SB Tech Services mb: +61 437 286 200

_______________________________________________
Mailing list: https://launchpad.net/~kicad-developers
Post to     : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp