Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Simon Küppers]

For what is worth, I strongly support this standpoint! :-)


Am 06.10.2017 um 15:33 schrieb Oliver Walters:
> Wayne,
> 
> Thanks for the clarification - this discussion is getting a bit
> sidetracked I feel.
> 
> We will operate under the assumption that the footprints will be in a
> single repo, without using submodules.
> 
> Any tools that can or will be developed for managing KiCad libraries are
> a secondary requirement at this point.
> 
> Cheers,
> 
> Oliver
> 
> On Sat, Oct 7, 2017 at 12:20 AM, Wayne Stambaugh  > wrote:
> 
> On 10/6/2017 5:43 AM, Rene Pöschl wrote:
> > On 02/10/17 11:37, Simon Küppers wrote:
> >> I am no expert, but this question is out of the scope of this
> thread..
> >> For downloading, gut applies compression anyway.
> >>
> >> I respect the people having slow Internet lines.. However as shown a
> >> few posts backwards, the whole footprints and symbol library is like
> >> 100 megabytes without the 3d models. If think the benefit of a single
> >> repo outways the ability to download only a selection of libraries...
> >> By a LOT.
> >>
> > I agree with you here. Lets ignore the 3d model stuff and get back
> > talking about the footprint libs.
> > We are planning a massive library reorganization. (for the v5 release)
> > This will require a lot more footprint libs. If all these new libs are
> > all in a separate repo then this can not be done!
> > In addition to the new repos the old once still need to exist (and
> have
> > at least one footprint each) because otherwise the github plugin will
> > stop working for everyone who has this old repo in their fp-lib-table.
> >
> > **We library managers require whatever lib distribution is used to
> > support one repo that holds all footprint libs.** (one repo that
> has the
> > pretty folders as subdirectorys)
> > To be honest i do not care what will be used as long as this
> requirement
> > is fulfilled.
> >
> > We would really need a decision soon. Are you guys prepared to help us
> > out here such that we can move on with the lib reorganization? (The
> > details of how you do it can be discussed later.)
> >
> 
> I thought I already addressed this issue but I will reiterate.  I am
> fine with the footprint library reorganization as long as the existing
> footprint library repos remain in tact for existing users.  These
> libraries do not need to be updated but we cannot remove them.  The
> other thing that will have to happen is that our package devs will have
> to buy into packaging the new footprint library repo as part of the
> install packages.  We will have to agree on an install path (
> ${CMAKE_INSTALL_PREFIX}/share/kicad/footprints seems logical) and make
> the default fp-lib-table file use the local install path with the KiCad
> plugin instead of the github plugin fp-lib-table.  I don't think this
> will be a major issue.  Does anyone have any major objections to this?
> If not, we will make this part of the stable 5 release.
> 
> Cheers,
> 
> Wayne
> 
> ___
> Mailing list: https://launchpad.net/~kicad-developers
> 
> Post to : kicad-developers@lists.launchpad.net
> 
> Unsubscribe : https://launchpad.net/~kicad-developers
> 
> More help   : https://help.launchpad.net/ListHelp
> 
> 
> 
> 
> 
> ___
> Mailing list: https://launchpad.net/~kicad-developers
> Post to : kicad-developers@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
> 

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Wayne Stambaugh]

On 10/6/2017 9:33 AM, Oliver Walters wrote:
> Wayne,
> 
> Thanks for the clarification - this discussion is getting a bit
> sidetracked I feel.
> 
> We will operate under the assumption that the footprints will be in a
> single repo, without using submodules.
> 
> Any tools that can or will be developed for managing KiCad libraries are
> a secondary requirement at this point.

Yes, this is a separate discussion and will not be part of the stable 5
release.

> 
> Cheers,
> 
> Oliver
> 
> On Sat, Oct 7, 2017 at 12:20 AM, Wayne Stambaugh  > wrote:
> 
> On 10/6/2017 5:43 AM, Rene Pöschl wrote:
> > On 02/10/17 11:37, Simon Küppers wrote:
> >> I am no expert, but this question is out of the scope of this
> thread..
> >> For downloading, gut applies compression anyway.
> >>
> >> I respect the people having slow Internet lines.. However as shown a
> >> few posts backwards, the whole footprints and symbol library is like
> >> 100 megabytes without the 3d models. If think the benefit of a single
> >> repo outways the ability to download only a selection of libraries...
> >> By a LOT.
> >>
> > I agree with you here. Lets ignore the 3d model stuff and get back
> > talking about the footprint libs.
> > We are planning a massive library reorganization. (for the v5 release)
> > This will require a lot more footprint libs. If all these new libs are
> > all in a separate repo then this can not be done!
> > In addition to the new repos the old once still need to exist (and
> have
> > at least one footprint each) because otherwise the github plugin will
> > stop working for everyone who has this old repo in their fp-lib-table.
> >
> > **We library managers require whatever lib distribution is used to
> > support one repo that holds all footprint libs.** (one repo that
> has the
> > pretty folders as subdirectorys)
> > To be honest i do not care what will be used as long as this
> requirement
> > is fulfilled.
> >
> > We would really need a decision soon. Are you guys prepared to help us
> > out here such that we can move on with the lib reorganization? (The
> > details of how you do it can be discussed later.)
> >
> 
> I thought I already addressed this issue but I will reiterate.  I am
> fine with the footprint library reorganization as long as the existing
> footprint library repos remain in tact for existing users.  These
> libraries do not need to be updated but we cannot remove them.  The
> other thing that will have to happen is that our package devs will have
> to buy into packaging the new footprint library repo as part of the
> install packages.  We will have to agree on an install path (
> ${CMAKE_INSTALL_PREFIX}/share/kicad/footprints seems logical) and make
> the default fp-lib-table file use the local install path with the KiCad
> plugin instead of the github plugin fp-lib-table.  I don't think this
> will be a major issue.  Does anyone have any major objections to this?
> If not, we will make this part of the stable 5 release.
> 
> Cheers,
> 
> Wayne
> 
> ___
> Mailing list: https://launchpad.net/~kicad-developers
> 
> Post to     : kicad-developers@lists.launchpad.net
> 
> Unsubscribe : https://launchpad.net/~kicad-developers
> 
> More help   : https://help.launchpad.net/ListHelp
> 
> 
> 


___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Oliver Walters]

Wayne,

Thanks for the clarification - this discussion is getting a bit sidetracked
I feel.

We will operate under the assumption that the footprints will be in a
single repo, without using submodules.

Any tools that can or will be developed for managing KiCad libraries are a
secondary requirement at this point.

Cheers,

Oliver

On Sat, Oct 7, 2017 at 12:20 AM, Wayne Stambaugh 
wrote:

> On 10/6/2017 5:43 AM, Rene Pöschl wrote:
> > On 02/10/17 11:37, Simon Küppers wrote:
> >> I am no expert, but this question is out of the scope of this thread..
> >> For downloading, gut applies compression anyway.
> >>
> >> I respect the people having slow Internet lines.. However as shown a
> >> few posts backwards, the whole footprints and symbol library is like
> >> 100 megabytes without the 3d models. If think the benefit of a single
> >> repo outways the ability to download only a selection of libraries...
> >> By a LOT.
> >>
> > I agree with you here. Lets ignore the 3d model stuff and get back
> > talking about the footprint libs.
> > We are planning a massive library reorganization. (for the v5 release)
> > This will require a lot more footprint libs. If all these new libs are
> > all in a separate repo then this can not be done!
> > In addition to the new repos the old once still need to exist (and have
> > at least one footprint each) because otherwise the github plugin will
> > stop working for everyone who has this old repo in their fp-lib-table.
> >
> > **We library managers require whatever lib distribution is used to
> > support one repo that holds all footprint libs.** (one repo that has the
> > pretty folders as subdirectorys)
> > To be honest i do not care what will be used as long as this requirement
> > is fulfilled.
> >
> > We would really need a decision soon. Are you guys prepared to help us
> > out here such that we can move on with the lib reorganization? (The
> > details of how you do it can be discussed later.)
> >
>
> I thought I already addressed this issue but I will reiterate.  I am
> fine with the footprint library reorganization as long as the existing
> footprint library repos remain in tact for existing users.  These
> libraries do not need to be updated but we cannot remove them.  The
> other thing that will have to happen is that our package devs will have
> to buy into packaging the new footprint library repo as part of the
> install packages.  We will have to agree on an install path (
> ${CMAKE_INSTALL_PREFIX}/share/kicad/footprints seems logical) and make
> the default fp-lib-table file use the local install path with the KiCad
> plugin instead of the github plugin fp-lib-table.  I don't think this
> will be a major issue.  Does anyone have any major objections to this?
> If not, we will make this part of the stable 5 release.
>
> Cheers,
>
> Wayne
>
> ___
> Mailing list: https://launchpad.net/~kicad-developers
> Post to : kicad-developers@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Wayne Stambaugh]

On 10/6/2017 5:43 AM, Rene Pöschl wrote:
> On 02/10/17 11:37, Simon Küppers wrote:
>> I am no expert, but this question is out of the scope of this thread..
>> For downloading, gut applies compression anyway.
>>
>> I respect the people having slow Internet lines.. However as shown a
>> few posts backwards, the whole footprints and symbol library is like
>> 100 megabytes without the 3d models. If think the benefit of a single
>> repo outways the ability to download only a selection of libraries...
>> By a LOT.
>>
> I agree with you here. Lets ignore the 3d model stuff and get back
> talking about the footprint libs.
> We are planning a massive library reorganization. (for the v5 release)
> This will require a lot more footprint libs. If all these new libs are
> all in a separate repo then this can not be done!
> In addition to the new repos the old once still need to exist (and have
> at least one footprint each) because otherwise the github plugin will
> stop working for everyone who has this old repo in their fp-lib-table.
> 
> **We library managers require whatever lib distribution is used to
> support one repo that holds all footprint libs.** (one repo that has the
> pretty folders as subdirectorys)
> To be honest i do not care what will be used as long as this requirement
> is fulfilled.
> 
> We would really need a decision soon. Are you guys prepared to help us
> out here such that we can move on with the lib reorganization? (The
> details of how you do it can be discussed later.)
> 

I thought I already addressed this issue but I will reiterate.  I am
fine with the footprint library reorganization as long as the existing
footprint library repos remain in tact for existing users.  These
libraries do not need to be updated but we cannot remove them.  The
other thing that will have to happen is that our package devs will have
to buy into packaging the new footprint library repo as part of the
install packages.  We will have to agree on an install path (
${CMAKE_INSTALL_PREFIX}/share/kicad/footprints seems logical) and make
the default fp-lib-table file use the local install path with the KiCad
plugin instead of the github plugin fp-lib-table.  I don't think this
will be a major issue.  Does anyone have any major objections to this?
If not, we will make this part of the stable 5 release.

Cheers,

Wayne

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Rene Pöschl]

On 02/10/17 11:37, Simon Küppers wrote:
I am no expert, but this question is out of the scope of this thread.. 
For downloading, gut applies compression anyway.


I respect the people having slow Internet lines.. However as shown a 
few posts backwards, the whole footprints and symbol library is like 
100 megabytes without the 3d models. If think the benefit of a single 
repo outways the ability to download only a selection of libraries... 
By a LOT.


I agree with you here. Lets ignore the 3d model stuff and get back 
talking about the footprint libs.

We are planning a massive library reorganization. (for the v5 release)
This will require a lot more footprint libs. If all these new libs are 
all in a separate repo then this can not be done!
In addition to the new repos the old once still need to exist (and have 
at least one footprint each) because otherwise the github plugin will 
stop working for everyone who has this old repo in their fp-lib-table.


**We library managers require whatever lib distribution is used to 
support one repo that holds all footprint libs.** (one repo that has the 
pretty folders as subdirectorys)
To be honest i do not care what will be used as long as this requirement 
is fulfilled.


We would really need a decision soon. Are you guys prepared to help us 
out here such that we can move on with the lib reorganization? (The 
details of how you do it can be discussed later.)


___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Andrey Kuznetsov]

Sorry, I was only talking about wrapping the 3D files which can be 100KB to
10MB large.
The other files are small enough.

On Mon, Oct 2, 2017 at 2:38 AM Simon Küppers  wrote:

> I am no expert, but this question is out of the scope of this thread.. For
> downloading, gut applies compression anyway.
>
> I respect the people having slow Internet lines.. However as shown a few
> posts backwards, the whole footprints and symbol library is like 100
> megabytes without the 3d models. If think the benefit of a single repo
> outways the ability to download only a selection of libraries... By a LOT.
>
>
>
> Am 2. Oktober 2017 10:22:10 MESZ schrieb Andrey Kuznetsov <
> kandre...@gmail.com>:
>>
>> Is it possible to keep the 3d models ZIPed or RARed on disk and KiCAD
>> unpacks them on the fly as needed/used during the session/etc?
>>
>> I am seeing 10x reduction in size when I pack the files whether they are
>> 7.5MB or 120KB, both were reduced to 750KB and 10KB respectively.
>> That's a lot of space wasted if we're thinking of the poor designers with
>> limited disk space.
>>
>> On Mon, Oct 2, 2017 at 1:03 AM, Carsten Schoenert <
>> c.schoen...@t-online.de> wrote:
>>
>>> Am 02.10.2017 um 06:14 schrieb David Godfrey:
>>> > Bernhard hit the nail on the head here.
>>> > For normal Users, ALL of the git functionality should be hidden behind
>>> > basic KiCad GUI features.
>>>
>>> A "normal" user doesn't need any git functions. He expects to have a
>>> working solution if he is using KiCad or $whatever software. The tricky
>>> part is on the software developers side, they need to take care about
>>> full functional additional components for the normal users.
>>>
>>> > However, for Users and Librarians that want to manage, add, edit at
>>> > least a basic knowledge of whatever tooling is used behind the scenes
>>> is
>>> > a HARD REQUIREMENT.
>>>
>>> Agreed.
>>> But such things are additional extras on the current situation. I guess
>>> the intent of this whole thread was to improve the current situation on
>>> the library handling inside KiCad. I think this should be focused on
>>> first as this increases the usability on the user side significantly.
>>>
>>> > These days git is probably one of the best documented, and most well
>>> > supported in the greater community.
>>> > That alone makes it a very good choice of backend.
>>> >
>>> > Handling of submodules can be slightly tricky, but a few simple helper
>>> > scripts (for LedgerSMB project we use a Makefile with a few targets
>>> such
>>> > as "submodules" which updates all submodules to the current repo head's
>>> > commit references to them)
>>> Mhh, I never have seen that any body is really happy about git
>>> submodules as they are always problematic. The reasons for this are
>>> already written here in this thread.
>>>
>>> I always look at the Linux kernel development model which is quite
>>> larger and bigger than the KiCad project.
>>> All parts in the development there don't use git submoduls for good
>>> reasons. All people involved always use the full tree. Sorry, I don't
>>> see a real need and gain for using git submoduls. And even if you have
>>> some scripting on top you need to teach the people how to use this. That
>>> is *always* overhead I'd avoid.
>>>
>>> As written here also, a complete git repository about all of the
>>> schematics with a stable and development branch and tagged releases
>>> would be fine and enough. The l10n and documentation part is already
>>> using this model.
>>>
>>> --
>>> Regards
>>> Carsten Schoenert
>>>
>>> ___
>>> Mailing list: https://launchpad.net/~kicad-developers
>>> Post to : kicad-developers@lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~kicad-developers
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>
>>
>>
>> --
>> Remember The Past, Live The Present, Change The Future
>> Those who look only to the past or the present are certain to miss the
>> future [JFK]
>>
>> kandre...@gmail.com
>> Live Long and Prosper,
>> Andrey
>>
> ___
> Mailing list: https://launchpad.net/~kicad-developers
> Post to : kicad-developers@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>
-- 
Remember The Past, Live The Present, Change The Future
Those who look only to the past or the present are certain to miss the
future [JFK]

kandre...@gmail.com
Live Long and Prosper,
Andrey
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Simon Küppers]

I am no expert, but this question is out of the scope of this thread.. For 
downloading, gut applies compression anyway. 

I respect the people having slow Internet lines.. However as shown a few posts 
backwards, the whole footprints and symbol library is like 100 megabytes 
without the 3d models. If think the benefit of a single repo outways the 
ability to download only a selection of libraries... By a LOT. 



Am 2. Oktober 2017 10:22:10 MESZ schrieb Andrey Kuznetsov :
>Is it possible to keep the 3d models ZIPed or RARed on disk and KiCAD
>unpacks them on the fly as needed/used during the session/etc?
>
>I am seeing 10x reduction in size when I pack the files whether they
>are
>7.5MB or 120KB, both were reduced to 750KB and 10KB respectively.
>That's a lot of space wasted if we're thinking of the poor designers
>with
>limited disk space.
>
>On Mon, Oct 2, 2017 at 1:03 AM, Carsten Schoenert
>
>wrote:
>
>> Am 02.10.2017 um 06:14 schrieb David Godfrey:
>> > Bernhard hit the nail on the head here.
>> > For normal Users, ALL of the git functionality should be hidden
>behind
>> > basic KiCad GUI features.
>>
>> A "normal" user doesn't need any git functions. He expects to have a
>> working solution if he is using KiCad or $whatever software. The
>tricky
>> part is on the software developers side, they need to take care about
>> full functional additional components for the normal users.
>>
>> > However, for Users and Librarians that want to manage, add, edit at
>> > least a basic knowledge of whatever tooling is used behind the
>scenes is
>> > a HARD REQUIREMENT.
>>
>> Agreed.
>> But such things are additional extras on the current situation. I
>guess
>> the intent of this whole thread was to improve the current situation
>on
>> the library handling inside KiCad. I think this should be focused on
>> first as this increases the usability on the user side significantly.
>>
>> > These days git is probably one of the best documented, and most
>well
>> > supported in the greater community.
>> > That alone makes it a very good choice of backend.
>> >
>> > Handling of submodules can be slightly tricky, but a few simple
>helper
>> > scripts (for LedgerSMB project we use a Makefile with a few targets
>such
>> > as "submodules" which updates all submodules to the current repo
>head's
>> > commit references to them)
>> Mhh, I never have seen that any body is really happy about git
>> submodules as they are always problematic. The reasons for this are
>> already written here in this thread.
>>
>> I always look at the Linux kernel development model which is quite
>> larger and bigger than the KiCad project.
>> All parts in the development there don't use git submoduls for good
>> reasons. All people involved always use the full tree. Sorry, I don't
>> see a real need and gain for using git submoduls. And even if you
>have
>> some scripting on top you need to teach the people how to use this.
>That
>> is *always* overhead I'd avoid.
>>
>> As written here also, a complete git repository about all of the
>> schematics with a stable and development branch and tagged releases
>> would be fine and enough. The l10n and documentation part is already
>> using this model.
>>
>> --
>> Regards
>> Carsten Schoenert
>>
>> ___
>> Mailing list: https://launchpad.net/~kicad-developers
>> Post to : kicad-developers@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~kicad-developers
>> More help   : https://help.launchpad.net/ListHelp
>>
>
>
>
>-- 
>Remember The Past, Live The Present, Change The Future
>Those who look only to the past or the present are certain to miss the
>future [JFK]
>
>kandre...@gmail.com
>Live Long and Prosper,
>Andrey
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Andrey Kuznetsov]

Is it possible to keep the 3d models ZIPed or RARed on disk and KiCAD
unpacks them on the fly as needed/used during the session/etc?

I am seeing 10x reduction in size when I pack the files whether they are
7.5MB or 120KB, both were reduced to 750KB and 10KB respectively.
That's a lot of space wasted if we're thinking of the poor designers with
limited disk space.

On Mon, Oct 2, 2017 at 1:03 AM, Carsten Schoenert 
wrote:

> Am 02.10.2017 um 06:14 schrieb David Godfrey:
> > Bernhard hit the nail on the head here.
> > For normal Users, ALL of the git functionality should be hidden behind
> > basic KiCad GUI features.
>
> A "normal" user doesn't need any git functions. He expects to have a
> working solution if he is using KiCad or $whatever software. The tricky
> part is on the software developers side, they need to take care about
> full functional additional components for the normal users.
>
> > However, for Users and Librarians that want to manage, add, edit at
> > least a basic knowledge of whatever tooling is used behind the scenes is
> > a HARD REQUIREMENT.
>
> Agreed.
> But such things are additional extras on the current situation. I guess
> the intent of this whole thread was to improve the current situation on
> the library handling inside KiCad. I think this should be focused on
> first as this increases the usability on the user side significantly.
>
> > These days git is probably one of the best documented, and most well
> > supported in the greater community.
> > That alone makes it a very good choice of backend.
> >
> > Handling of submodules can be slightly tricky, but a few simple helper
> > scripts (for LedgerSMB project we use a Makefile with a few targets such
> > as "submodules" which updates all submodules to the current repo head's
> > commit references to them)
> Mhh, I never have seen that any body is really happy about git
> submodules as they are always problematic. The reasons for this are
> already written here in this thread.
>
> I always look at the Linux kernel development model which is quite
> larger and bigger than the KiCad project.
> All parts in the development there don't use git submoduls for good
> reasons. All people involved always use the full tree. Sorry, I don't
> see a real need and gain for using git submoduls. And even if you have
> some scripting on top you need to teach the people how to use this. That
> is *always* overhead I'd avoid.
>
> As written here also, a complete git repository about all of the
> schematics with a stable and development branch and tagged releases
> would be fine and enough. The l10n and documentation part is already
> using this model.
>
> --
> Regards
> Carsten Schoenert
>
> ___
> Mailing list: https://launchpad.net/~kicad-developers
> Post to : kicad-developers@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>



-- 
Remember The Past, Live The Present, Change The Future
Those who look only to the past or the present are certain to miss the
future [JFK]

kandre...@gmail.com
Live Long and Prosper,
Andrey
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Carsten Schoenert]

Am 02.10.2017 um 06:14 schrieb David Godfrey:
> Bernhard hit the nail on the head here.
> For normal Users, ALL of the git functionality should be hidden behind
> basic KiCad GUI features.

A "normal" user doesn't need any git functions. He expects to have a
working solution if he is using KiCad or $whatever software. The tricky
part is on the software developers side, they need to take care about
full functional additional components for the normal users.

> However, for Users and Librarians that want to manage, add, edit at
> least a basic knowledge of whatever tooling is used behind the scenes is
> a HARD REQUIREMENT.

Agreed.
But such things are additional extras on the current situation. I guess
the intent of this whole thread was to improve the current situation on
the library handling inside KiCad. I think this should be focused on
first as this increases the usability on the user side significantly.

> These days git is probably one of the best documented, and most well
> supported in the greater community.
> That alone makes it a very good choice of backend.
> 
> Handling of submodules can be slightly tricky, but a few simple helper
> scripts (for LedgerSMB project we use a Makefile with a few targets such
> as "submodules" which updates all submodules to the current repo head's
> commit references to them)
Mhh, I never have seen that any body is really happy about git
submodules as they are always problematic. The reasons for this are
already written here in this thread.

I always look at the Linux kernel development model which is quite
larger and bigger than the KiCad project.
All parts in the development there don't use git submoduls for good
reasons. All people involved always use the full tree. Sorry, I don't
see a real need and gain for using git submoduls. And even if you have
some scripting on top you need to teach the people how to use this. That
is *always* overhead I'd avoid.

As written here also, a complete git repository about all of the
schematics with a stable and development branch and tagged releases
would be fine and enough. The l10n and documentation part is already
using this model.

-- 
Regards
Carsten Schoenert

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Thomas Kindler]

On 02.10.2017 07:00, David Godfrey wrote:
> [..]

> Don't Forget that not all of the world's internet connections are FAST.
> While it might seem a good idea from the librarians perspective to have
> everything in a single repo, IT ISN'T a good idea at all if you happen to be 
> on
> a slow internet connection.
> 
> When I say slow, I'm talking common ADSL speeds here in Australia for example.
> They are often 8mbit/s or less.
> 

8 MBit/s is not too shabby. You could download 3.6 gigabytes in one hour.

> [..]

> If the entire lib ended up being ~ 3G (a figure mentioned in other posts) that
> would Take an Unacceptably long time to download.

If you are refering to my post from 2017-09-23:

* The worst-case download size would be 650 MB for everything (history, 3D
models, etc).

* 3.8 Gigabyte is the on-disk size after decompression

As said, 90% of that size comes from generated 3D models. It might make sense to
keep them in a separate repo for people with slow computers or internet 
connections.


Here are the .git directory (i.e. download with full history) sizes:

  kicad-library (with 3D models) :  650 MB

  kicad-library (without 3D models)  : < 22 MB  (estimated)
  kicad-footprints (with all 88 submodules)  :   51 MB
  kicad-packages3D-source:   16 MB


> For this reason I would STRONGLY advise libraries remain broken up into
> manageable sized repositories

The current situation is madness. The footprints are broken into tiny,
unmanageable chunks for no apparent reason at all.

The only chunk to separate out that makes sense is the 3D models.

Even with the best scripts and tools, git submodules are a pain to work with and
a *BIG* turn-down for contributors.

> Also, keep in mind that a repository never gets smaller, it only grows as
> changes are made.
> 

Yes, but it will grow *very* slowly. Footprints and Library symbols are text
files that compress and diff great.

The generated 3D models might be another story.

> [..]

best regards,
-- 
Thomas Kindler

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[David Godfrey]

  
  
Hi Oliver,


  An important note before I get to other responses.
  
  Don't Forget that not all of the world's internet connections are
  FAST.
  While it might seem a good idea from the librarians perspective to
  have everything in a single repo, IT ISN'T a good idea at all if
  you happen to be on a slow internet connection.
  
  When I say slow, I'm talking common ADSL speeds here in Australia
  for example.
  They are often 8mbit/s or less.
  
  Even with the "National Broadband Network" rollout of Fibre, I
  know people that effectively suffered a net reduction of speed
  compared to their old ADSL connection. (Yeah, crazy, but it's
  true)
  
  If the entire lib ended up being ~ 3G (a figure mentioned in other
  posts) that would Take an Unacceptably long time to download.
  
  For this reason I would STRONGLY advise libraries remain broken up
  into manageable sized repositories
  Also, keep in mind that a repository never gets smaller, it only
  grows as changes are made. 
  

On 28/09/17 19:29, Oliver Walters
  wrote:


  Simon, David,


Thanks for the synopsis of your idea David, there are
  certainly some advantages to using the submodule approach.


Is it github compatible?
 
Simon, yes git-submodule is compatible with github. I have
  a test repository here - https://github.com/kicad/kicad-footprints
  - this contains a submodule for each of the current footprint
  directories.



Does it simplify the maintenance of
the kicad library by the librarians?


Compared to having all the .pretty folders as
  subdirectories of a single repo, no. Reorganising libraries or
  adding new ones is tedious, and for each of the 100+ .pretty
  repos, we (the librarians) have to manage:


* Pull requests and isues
* CI tools (e.g. Travis KLC scripts)
* Updating each library when new requirements are enacted
  (e.g. KLC)
* Ensure that changes in one repo do not conflict with a
  different repo
* Handle reclassification or moving of footprints between
  repos


  

Agreed, submodules may well add some management overhead, although
most of the items in the list you gave are still relevant even if
all libs are in a single repo.

One possibly significant advantage of using a submodule structure is
the ease of inclusion of libraries that are not part of the "core"
project managed set.
The aids in three ways.
 * Trivial inclusion of Vendor provided libraries in the Core Set,
by simply including an extra submodule
 * Easy addition of site or project specific libs by having a new
top level repo, that includes the Core repo, and additional libs
 * Easy reduction in the range of libs available to a team by having
a fork of the Core repo with limited defined submodules

NOTE: Submodules are not the only way to achieve these goals, but in
many ways, as long as the appropriate helper scripts etc are  put in
place, submodules are likely the easiest to maintain in the long
run.
Potentially, the entire available-to-the-current-project library
mechanism could be designed to use a local git repo, that simply
includes submodules as required.

  
It is also quite possible for submodules to become out of
  sync. The link that Simon posted has some a good example of
  how this can occur (detached head). I have some projects at
  work that contain about 5 submodules and it rapidly becomes
  difficult to deal with even for a competent user of Git.
  

As I've mentioned elsewhere, 99% of the problems with submodules
becoming out of sync can be resolved with a few simple helper
scripts, potentially in the form of makefile targets.

  


It becomes harder for users, too. KiCad users cannot, in
  the general case, be classified as competent users of git.
  Currently, users who wish to contribute to the footprint
  libraries have a hard time. For each library they want to
  contribute to, they have to fork and manage another
  repository. Personally I have done this for 50 separate
  repositories. 

  

Ideally, yes, forking of repo's is the best way to manage user
contributions.
To a large extent, I'd have thought most users are likely to only
contribute to a small number of libraries directly.
So that may not be a significant burden.

However, There is an alternative way to handle contributions,
assuming we manage it all from a

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[David Godfrey]

  
  
On 29/09/17 01:53, Bernhard Stegmaier wrote:

  
  Just my 2 cents… I have seen some projects using submodules (open
  source and in my company).
  I have seen lots of problems and complaints (…
doesn’t compile …) just because normal users or even normal
developers (being no git experts) didn’t sync all submodules
correctly.
  Thinking of how many problems are already seen now
by users which are not used to version control systems (because
they just don’t understand what’s happening under the hood) I
wouldn’t want to force them to use git submodules… at least not
as long as this is not completely hidden by some users friendly
and dead simple KiCad GUI.
  
  

Bernhard hit the nail on the head here.
For normal Users, ALL of the git functionality should be hidden
behind basic KiCad GUI features.

However, for Users and Librarians that want to manage, add, edit at
least a basic knowledge of whatever tooling is used behind the
scenes is a HARD REQUIREMENT.

These days git is probably one of the best documented, and most well
supported in the greater community.
That alone makes it a very good choice of backend.

Handling of submodules can be slightly tricky, but a few simple
helper scripts (for LedgerSMB project we use a Makefile with a few
targets such as "submodules" which updates all submodules to the
current repo head's commit references to them)


  
  
  Regards,
  Bernhard
  

  
On 28. Sep 2017, at 13:29, Oliver Walters 
  wrote:


  Simon, David,


Thanks for the synopsis of your idea
  David, there are certainly some advantages to using
  the submodule approach.


Is it github
compatible?
 
Simon, yes git-submodule is compatible
  with github. I have a test repository here - https://github.com/kicad/kicad-footprints - this contains a
  submodule for each of the current footprint
  directories.



Does it simplify
the maintenance of the kicad library by the
librarians?


Compared to having all the .pretty folders
  as subdirectories of a single repo, no. Reorganising
  libraries or adding new ones is tedious, and for each
  of the 100+ .pretty repos, we (the librarians) have to
  manage:


* Pull requests and isues
* CI tools (e.g. Travis KLC scripts)
* Updating each library when new
  requirements are enacted (e.g. KLC)
* Ensure that changes in one repo do not
  conflict with a different repo
* Handle reclassification or moving of
  footprints between repos


It is also quite possible for submodules
  to become out of sync. The link that Simon posted has
  some a good example of how this can occur (detached
  head). I have some projects at work that contain about
  5 submodules and it rapidly becomes difficult to deal
  with even for a competent user of Git.


It becomes harder for users, too. KiCad
  users cannot, in the general case, be classified as
  competent users of git. Currently, users who wish to
  contribute to the footprint libraries have a hard
  time. For each library they want to contribute to,
  they have to fork and manage another repository.
  Personally I have done this for 50 separate
  repositories. 


Further, once changes are made to each
  submodule, the master repository then needs to be
  updated otherwise it's not up to date with the latest
  changes. Perhaps we can use git hooks here, but this
  is just further complication!


Let's also consider consistency. The
  footprint libraries are (currently) split into
  separate repos. However the symbol libraries and 3d
  model libraries are not. Should we split them too? Now
  we have 300+ repositories to manage.

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Diego Herranz]

I completely understand Oliver's view on maintaining every individual repo
even if they are submodules of a kicad-footprint repo.

Even from an a user perspective (which is clearly easier than the
maintainer one), I like knowing what progress is being made, so I'd like to
subscribe for updates of a single repo on github, to follow commits, pull
requests, etc. There's no way I will subscribe to every repo and what's
most difficult, checking to make sure that I subscribe for new repos as
they are created too.

I have basically no say here, but I'd go for a single repo and no
submodules.

Thanks,
Diego

On Thu, Sep 28, 2017 at 6:53 PM, Bernhard Stegmaier  wrote:

> Just my 2 cents… I have seen some projects using submodules (open source
> and in my company).
> I have seen lots of problems and complaints (… doesn’t compile …) just
> because normal users or even normal developers (being no git experts)
> didn’t sync all submodules correctly.
> Thinking of how many problems are already seen now by users which are not
> used to version control systems (because they just don’t understand what’s
> happening under the hood) I wouldn’t want to force them to use git
> submodules… at least not as long as this is not completely hidden by some
> users friendly and dead simple KiCad GUI.
>
>
> Regards,
> Bernhard
>
> On 28. Sep 2017, at 13:29, Oliver Walters 
> wrote:
>
> Simon, David,
>
> Thanks for the synopsis of your idea David, there are certainly some
> advantages to using the submodule approach.
>
> Is it github compatible?
>
>
> Simon, yes git-submodule is compatible with github. I have a test
> repository here - https://github.com/kicad/kicad-footprints - this
> contains a submodule for each of the current footprint directories.
>
> Does it simplify the maintenance of the kicad library by the librarians?
>
>
> Compared to having all the .pretty folders as subdirectories of a single
> repo, no. Reorganising libraries or adding new ones is tedious, and for
> each of the 100+ .pretty repos, we (the librarians) have to manage:
>
> * Pull requests and isues
> * CI tools (e.g. Travis KLC scripts)
> * Updating each library when new requirements are enacted (e.g. KLC)
> * Ensure that changes in one repo do not conflict with a different repo
> * Handle reclassification or moving of footprints between repos
>
> It is also quite possible for submodules to become out of sync. The link
> that Simon posted has some a good example of how this can occur (detached
> head). I have some projects at work that contain about 5 submodules and it
> rapidly becomes difficult to deal with even for a competent user of Git.
>
> It becomes harder for users, too. KiCad users cannot, in the general case,
> be classified as competent users of git. Currently, users who wish to
> contribute to the footprint libraries have a hard time. For each library
> they want to contribute to, they have to fork and manage another
> repository. Personally I have done this for 50 separate repositories.
>
> Further, once changes are made to each submodule, the master repository
> then needs to be updated otherwise it's not up to date with the latest
> changes. Perhaps we can use git hooks here, but this is just further
> complication!
>
> Let's also consider consistency. The footprint libraries are (currently)
> split into separate repos. However the symbol libraries and 3d model
> libraries are not. Should we split them too? Now we have 300+ repositories
> to manage.
>
> Let's not make this an academic problem and say "but it's possible to
> manage with this complex set of tools"! From the perspective of a
> contributor, and as the person who has to manage the libraries, the simpler
> the better.
>
> Oliver
>
>
>
> On Thu, Sep 28, 2017 at 7:25 PM, Simon Küppers 
> wrote:
>
>> Thanks for your detailed description. I think it is a nice way to go.
>> However two remarks:
>> Does it simplify the maintenance of the kicad library by the librarians?
>> Is it github compatible? If not, we would need to find another platform to
>> host the libraries.
>>
>> If I look for git submodule, there is a lot of different opinions on why
>> you should or should not use it. (e.g. https://www.google.de/amp/s/co
>> dingkilledthecat.wordpress.com/2012/04/28/why-your-company-
>> shouldnt-use-git-submodules/amp/). Does it apply in our case? What about
>> git subtree?
>>
>> Best regards
>> Simon
>>
>> Am 28. September 2017 02:37:09 MESZ schrieb David Godfrey <
>> i...@sbts.com.au>:
>>
>>> Hi All
>>>
>>> I have to agree with some of the other posters, why not use git as it
>>> was intended?
>>>
>>> Specifically I think the following makes sense.
>>>
>>> - Keep the multiple repositories, this helps when a company only wants
>>> to make certain sets of libs available to it's staff
>>>
>>> - Have a master repository that includes all other repositories as
>>> submodules.
>>>
>>> - Have branches that are matched to kicad versions. This allows
>>> footprint changes in a version safe way.

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Bernhard Stegmaier]

Just my 2 cents… I have seen some projects using submodules (open source and in 
my company).
I have seen lots of problems and complaints (… doesn’t compile …) just because 
normal users or even normal developers (being no git experts) didn’t sync all 
submodules correctly.
Thinking of how many problems are already seen now by users which are not used 
to version control systems (because they just don’t understand what’s happening 
under the hood) I wouldn’t want to force them to use git submodules… at least 
not as long as this is not completely hidden by some users friendly and dead 
simple KiCad GUI.


Regards,
Bernhard

> On 28. Sep 2017, at 13:29, Oliver Walters  
> wrote:
> 
> Simon, David,
> 
> Thanks for the synopsis of your idea David, there are certainly some 
> advantages to using the submodule approach.
> 
> Is it github compatible?
>  
> Simon, yes git-submodule is compatible with github. I have a test repository 
> here - https://github.com/kicad/kicad-footprints 
>  - this contains a submodule for 
> each of the current footprint directories.
> 
> Does it simplify the maintenance of the kicad library by the librarians?
> 
> Compared to having all the .pretty folders as subdirectories of a single 
> repo, no. Reorganising libraries or adding new ones is tedious, and for each 
> of the 100+ .pretty repos, we (the librarians) have to manage:
> 
> * Pull requests and isues
> * CI tools (e.g. Travis KLC scripts)
> * Updating each library when new requirements are enacted (e.g. KLC)
> * Ensure that changes in one repo do not conflict with a different repo
> * Handle reclassification or moving of footprints between repos
> 
> It is also quite possible for submodules to become out of sync. The link that 
> Simon posted has some a good example of how this can occur (detached head). I 
> have some projects at work that contain about 5 submodules and it rapidly 
> becomes difficult to deal with even for a competent user of Git.
> 
> It becomes harder for users, too. KiCad users cannot, in the general case, be 
> classified as competent users of git. Currently, users who wish to contribute 
> to the footprint libraries have a hard time. For each library they want to 
> contribute to, they have to fork and manage another repository. Personally I 
> have done this for 50 separate repositories. 
> 
> Further, once changes are made to each submodule, the master repository then 
> needs to be updated otherwise it's not up to date with the latest changes. 
> Perhaps we can use git hooks here, but this is just further complication!
> 
> Let's also consider consistency. The footprint libraries are (currently) 
> split into separate repos. However the symbol libraries and 3d model 
> libraries are not. Should we split them too? Now we have 300+ repositories to 
> manage.
> 
> Let's not make this an academic problem and say "but it's possible to manage 
> with this complex set of tools"! From the perspective of a contributor, and 
> as the person who has to manage the libraries, the simpler the better. 
> 
> Oliver
> 
> 
> 
> On Thu, Sep 28, 2017 at 7:25 PM, Simon Küppers  > wrote:
> Thanks for your detailed description. I think it is a nice way to go. However 
> two remarks:
> Does it simplify the maintenance of the kicad library by the librarians? Is 
> it github compatible? If not, we would need to find another platform to host 
> the libraries. 
> 
> If I look for git submodule, there is a lot of different opinions on why you 
> should or should not use it. (e.g. 
> https://www.google.de/amp/s/codingkilledthecat.wordpress.com/2012/04/28/why-your-company-shouldnt-use-git-submodules/amp
>  
> /).
>  Does it apply in our case? What about git subtree? 
> 
> Best regards 
> Simon 
> 
> Am 28. September 2017 02:37:09 MESZ schrieb David Godfrey  >:
> Hi All
> 
> I have to agree with some of the other posters, why not use git as it 
> was intended?
> 
> Specifically I think the following makes sense.
> 
> - Keep the multiple repositories, this helps when a company only wants 
> to make certain sets of libs available to it's staff
> 
> - Have a master repository that includes all other repositories as 
> submodules.
> 
> - Have branches that are matched to kicad versions. This allows 
> footprint changes in a version safe way.
> 
> - Use standard "git clone" (initial download) and "git pull" (update) on 
> the main repo which provides the entire set of available libs without 
> actually downloading the content for all libs.
> 
> - When a specific lib is needed, do similar to what we do now, and use 
> "git submodule update --init --recursive $submoduleName" to just pull 
> that specific submodule
> 
> - Allow the "main" library repo URI to be altered. This enables a 
> companies fork of the repo to

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Miguel Angel Ajo Pelayo]

I'd try to start with the simpler structure of having all the libraries as
subdirectories of the same git repo.

That doesn't mean that we could eventually add support for submodules, (for
anyone that wants to make use of it)...

But I agree that from the point of view of the librarians it's a mess to
have everything so scattered.


On Thu, Sep 28, 2017 at 1:29 PM, Oliver Walters <
oliver.henry.walt...@gmail.com> wrote:

> Simon, David,
>
> Thanks for the synopsis of your idea David, there are certainly some
> advantages to using the submodule approach.
>
> Is it github compatible?
>
>
> Simon, yes git-submodule is compatible with github. I have a test
> repository here - https://github.com/kicad/kicad-footprints - this
> contains a submodule for each of the current footprint directories.
>
> Does it simplify the maintenance of the kicad library by the librarians?
>
>
> Compared to having all the .pretty folders as subdirectories of a single
> repo, no. Reorganising libraries or adding new ones is tedious, and for
> each of the 100+ .pretty repos, we (the librarians) have to manage:
>
> * Pull requests and isues
> * CI tools (e.g. Travis KLC scripts)
> * Updating each library when new requirements are enacted (e.g. KLC)
> * Ensure that changes in one repo do not conflict with a different repo
> * Handle reclassification or moving of footprints between repos
>
> It is also quite possible for submodules to become out of sync. The link
> that Simon posted has some a good example of how this can occur (detached
> head). I have some projects at work that contain about 5 submodules and it
> rapidly becomes difficult to deal with even for a competent user of Git.
>
> It becomes harder for users, too. KiCad users cannot, in the general case,
> be classified as competent users of git. Currently, users who wish to
> contribute to the footprint libraries have a hard time. For each library
> they want to contribute to, they have to fork and manage another
> repository. Personally I have done this for 50 separate repositories.
>
> Further, once changes are made to each submodule, the master repository
> then needs to be updated otherwise it's not up to date with the latest
> changes. Perhaps we can use git hooks here, but this is just further
> complication!
>
> Let's also consider consistency. The footprint libraries are (currently)
> split into separate repos. However the symbol libraries and 3d model
> libraries are not. Should we split them too? Now we have 300+ repositories
> to manage.
>
> Let's not make this an academic problem and say "but it's possible to
> manage with this complex set of tools"! From the perspective of a
> contributor, and as the person who has to manage the libraries, the simpler
> the better.
>
> Oliver
>
>
>
> On Thu, Sep 28, 2017 at 7:25 PM, Simon Küppers 
> wrote:
>
>> Thanks for your detailed description. I think it is a nice way to go.
>> However two remarks:
>> Does it simplify the maintenance of the kicad library by the librarians?
>> Is it github compatible? If not, we would need to find another platform to
>> host the libraries.
>>
>> If I look for git submodule, there is a lot of different opinions on why
>> you should or should not use it. (e.g. https://www.google.de/amp/s/co
>> dingkilledthecat.wordpress.com/2012/04/28/why-your-company-
>> shouldnt-use-git-submodules/amp/). Does it apply in our case? What about
>> git subtree?
>>
>> Best regards
>> Simon
>>
>> Am 28. September 2017 02:37:09 MESZ schrieb David Godfrey <
>> i...@sbts.com.au>:
>>
>>> Hi All
>>>
>>> I have to agree with some of the other posters, why not use git as it
>>> was intended?
>>>
>>> Specifically I think the following makes sense.
>>>
>>> - Keep the multiple repositories, this helps when a company only wants
>>> to make certain sets of libs available to it's staff
>>>
>>> - Have a master repository that includes all other repositories as
>>> submodules.
>>>
>>> - Have branches that are matched to kicad versions. This allows
>>> footprint changes in a version safe way.
>>>
>>> - Use standard "git clone" (initial download) and "git pull" (update) on
>>> the main repo which provides the entire set of available libs without
>>> actually downloading the content for all libs.
>>>
>>> - When a specific lib is needed, do similar to what we do now, and use
>>> "git submodule update --init --recursive $submoduleName" to just pull
>>> that specific submodule
>>>
>>> - Allow the "main" library repo URI to be altered. This enables a
>>> companies fork of the repo to be used.
>>>
>>> - Allow the "main" library repo URI to be ANY valid git URI. That means
>>> a repo on a local fileserver rather than a http server can be used.
>>> Along with various security options etc.
>>>
>>> - Add a fairly simple scripted tool that's run "on release" to retrieve
>>> a README.md (an possibly a descriptor/index file) from each actual
>>> library repo and update those within the "main" repo. This removes the
>>> need to have the curr

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Oliver Walters]

Simon, David,

Thanks for the synopsis of your idea David, there are certainly some
advantages to using the submodule approach.

Is it github compatible?


Simon, yes git-submodule is compatible with github. I have a test
repository here - https://github.com/kicad/kicad-footprints - this contains
a submodule for each of the current footprint directories.

Does it simplify the maintenance of the kicad library by the librarians?


Compared to having all the .pretty folders as subdirectories of a single
repo, no. Reorganising libraries or adding new ones is tedious, and for
each of the 100+ .pretty repos, we (the librarians) have to manage:

* Pull requests and isues
* CI tools (e.g. Travis KLC scripts)
* Updating each library when new requirements are enacted (e.g. KLC)
* Ensure that changes in one repo do not conflict with a different repo
* Handle reclassification or moving of footprints between repos

It is also quite possible for submodules to become out of sync. The link
that Simon posted has some a good example of how this can occur (detached
head). I have some projects at work that contain about 5 submodules and it
rapidly becomes difficult to deal with even for a competent user of Git.

It becomes harder for users, too. KiCad users cannot, in the general case,
be classified as competent users of git. Currently, users who wish to
contribute to the footprint libraries have a hard time. For each library
they want to contribute to, they have to fork and manage another
repository. Personally I have done this for 50 separate repositories.

Further, once changes are made to each submodule, the master repository
then needs to be updated otherwise it's not up to date with the latest
changes. Perhaps we can use git hooks here, but this is just further
complication!

Let's also consider consistency. The footprint libraries are (currently)
split into separate repos. However the symbol libraries and 3d model
libraries are not. Should we split them too? Now we have 300+ repositories
to manage.

Let's not make this an academic problem and say "but it's possible to
manage with this complex set of tools"! From the perspective of a
contributor, and as the person who has to manage the libraries, the simpler
the better.

Oliver



On Thu, Sep 28, 2017 at 7:25 PM, Simon Küppers 
wrote:

> Thanks for your detailed description. I think it is a nice way to go.
> However two remarks:
> Does it simplify the maintenance of the kicad library by the librarians?
> Is it github compatible? If not, we would need to find another platform to
> host the libraries.
>
> If I look for git submodule, there is a lot of different opinions on why
> you should or should not use it. (e.g. https://www.google.de/amp/s/
> codingkilledthecat.wordpress.com/2012/04/28/why-your-
> company-shouldnt-use-git-submodules/amp/). Does it apply in our case?
> What about git subtree?
>
> Best regards
> Simon
>
> Am 28. September 2017 02:37:09 MESZ schrieb David Godfrey <
> i...@sbts.com.au>:
>
>> Hi All
>>
>> I have to agree with some of the other posters, why not use git as it
>> was intended?
>>
>> Specifically I think the following makes sense.
>>
>> - Keep the multiple repositories, this helps when a company only wants
>> to make certain sets of libs available to it's staff
>>
>> - Have a master repository that includes all other repositories as
>> submodules.
>>
>> - Have branches that are matched to kicad versions. This allows
>> footprint changes in a version safe way.
>>
>> - Use standard "git clone" (initial download) and "git pull" (update) on
>> the main repo which provides the entire set of available libs without
>> actually downloading the content for all libs.
>>
>> - When a specific lib is needed, do similar to what we do now, and use
>> "git submodule update --init --recursive $submoduleName" to just pull
>> that specific submodule
>>
>> - Allow the "main" library repo URI to be altered. This enables a
>> companies fork of the repo to be used.
>>
>> - Allow the "main" library repo URI to be ANY valid git URI. That means
>> a repo on a local fileserver rather than a http server can be used.
>> Along with various security options etc.
>>
>> - Add a fairly simple scripted tool that's run "on release" to retrieve
>> a README.md (an possibly a descriptor/index file) from each actual
>> library repo and update those within the "main" repo. This removes the
>> need to have the current case where manual edits are required to keep
>> the "main" repo in sync with what's available in the individual lib's.
>>
>> To add a new lib, it's then as simple as (from the main repo) doing
>> something like
>> - "git submodule add $URI"
>> - "./scripts/update-library-indexes.sh"
>> - "git commit -a -m $'added new module "modulename"\nupdated all module
>> descriptions and index'"
>> - "git push"
>>
>>
>> All of the above allows the Main repo to be forked by a company (or
>> individual) and have their own custom repo's added very easily.
>> It even allows lib

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Simon Küppers]

Thanks for your detailed description. I think it is a nice way to go. However 
two remarks:
Does it simplify the maintenance of the kicad library by the librarians? Is it 
github compatible? If not, we would need to find another platform to host the 
libraries. 

If I look for git submodule, there is a lot of different opinions on why you 
should or should not use it. (e.g. 
https://www.google.de/amp/s/codingkilledthecat.wordpress.com/2012/04/28/why-your-company-shouldnt-use-git-submodules/amp/).
 Does it apply in our case? What about git subtree? 

Best regards 
Simon 

Am 28. September 2017 02:37:09 MESZ schrieb David Godfrey :
>Hi All
>
>I have to agree with some of the other posters, why not use git as it 
>was intended?
>
>Specifically I think the following makes sense.
>
>- Keep the multiple repositories, this helps when a company only wants 
>to make certain sets of libs available to it's staff
>
>- Have a master repository that includes all other repositories as 
>submodules.
>
>- Have branches that are matched to kicad versions. This allows 
>footprint changes in a version safe way.
>
>- Use standard "git clone" (initial download) and "git pull" (update)
>on 
>the main repo which provides the entire set of available libs without 
>actually downloading the content for all libs.
>
>- When a specific lib is needed, do similar to what we do now, and use 
>"git submodule update --init --recursive $submoduleName" to just pull 
>that specific submodule
>
>- Allow the "main" library repo URI to be altered. This enables a 
>companies fork of the repo to be used.
>
>- Allow the "main" library repo URI to be ANY valid git URI. That means
>
>a repo on a local fileserver rather than a http server can be used. 
>Along with various security options etc.
>
>- Add a fairly simple scripted tool that's run "on release" to retrieve
>
>a README.md (an possibly a descriptor/index file) from each actual 
>library repo and update those within the "main" repo. This removes the 
>need to have the current case where manual edits are required to keep 
>the "main" repo in sync with what's available in the individual lib's.
>
>To add a new lib, it's then as simple as (from the main repo) doing 
>something like
>- "git submodule add $URI"
>- "./scripts/update-library-indexes.sh"
>- "git commit -a -m $'added new module "modulename"\nupdated all module
>
>descriptions and index'"
>- "git push"
>
>
>All of the above allows the Main repo to be forked by a company (or 
>individual) and have their own custom repo's added very easily.
>It even allows libraries to be easily excluded or replaced, all using 
>extremely well developed management tools.
>
>On a side note, git submodules are stored in the main repository 
>basically as URI that includes a commit reference.
>That means it's easy to specify a specific library version to include
>in 
>a specific branch/tag of the main repo.
>
>
>As for the person that said "git isn't available for windows", sorry, 
>that's bunkum.
>There are many many development environments out there that ONLY run on
>
>windows that have git integration.
>A quick google search for "git for windows" will show what you need to 
>know there.
>
>Finally, wrapping git to duplicate the effective API that's currently
>in 
>use should be relatively trivial, resulting in virtually no code
>changes 
>required to KICAD it's self.
>Assuming the existing functionality is cleanly wrapped by an API, the 
>only real change to KICAD would be the swapout of the module, and 
>addition of an easy way to change the URI
>
>
>I know I haven't covered everything in this email, but it should be a 
>good outline for further discussion
>
>
>On 22/09/17 09:21, Oliver Walters wrote:
>> Hi all,
>>
>> Ok, now that the website integration with the libraries is (pretty 
>> much) done, and the licensing issue seems to be sorted, there is one 
>> final puzzle piece to solve before I'm happy with the state of the 
>> libraries for a v5 release.
>>
>> *Goal: *Merge all footprint library repositories into a single repo
>to 
>> solve the ongoing dramas of maintaining 100+ repos.
>>
>> *Problem*: The *only* thing standing in the way of just doing this is
>
>> that some users like the GitHub plugin and previous instruction is 
>> that this functionality cannot be removed.
>>
>> The GitHub plugin functions by downloading a .zip file of a .pretty 
>> repo. If we merge all footprint libs into a single repo with multiple
>
>> subdirectories, this will not work anymore (as GitHub dosen't allow 
>> you to download a .zip of a single subdirectory).
>>
>> Merging the repos is the *right thing to do*. But how to proceed?
>>
>> *Options:*
>> *
>> *
>> /a) Drop github plugin feature, replace with library-download tool/
>>
>> I don't think it is a good idea to live-load library data from GitHub
>
>> (a lot of other users agree too). It's slow, and a waste of bandwidth
>
>> to re-download the libs all the time.
>>
>> We drop support for loading librarie

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Strontium]

+1 also
It allows a company to start with the libraries at some point and keep 
them consistent (by forking) and they can easily cherry pick changes or 
do wholesale merges with their changed libraries, as they desire.  And 
for a casual user who just uses things, "as is" it will just work.


On 28/09/17 09:47, Tiger12506 wrote:
+1 on this approach, to me this is the obvious way that it should be 
done. However, my opinion is not important, I'm not a contributor. :)


On 9/27/2017 8:37 PM, David Godfrey wrote:

Hi All

I have to agree with some of the other posters, why not use git as it 
was intended?


Specifically I think the following makes sense.

- Keep the multiple repositories, this helps when a company only 
wants to make certain sets of libs available to it's staff


- Have a master repository that includes all other repositories as 
submodules.


- Have branches that are matched to kicad versions. This allows 
footprint changes in a version safe way.


- Use standard "git clone" (initial download) and "git pull" (update) 
on the main repo which provides the entire set of available libs 
without actually downloading the content for all libs.


- When a specific lib is needed, do similar to what we do now, and 
use "git submodule update --init --recursive $submoduleName" to just 
pull that specific submodule


- Allow the "main" library repo URI to be altered. This enables a 
companies fork of the repo to be used.


- Allow the "main" library repo URI to be ANY valid git URI. That 
means a repo on a local fileserver rather than a http server can be 
used. Along with various security options etc.


- Add a fairly simple scripted tool that's run "on release" to 
retrieve a README.md (an possibly a descriptor/index file) from each 
actual library repo and update those within the "main" repo. This 
removes the need to have the current case where manual edits are 
required to keep the "main" repo in sync with what's available in the 
individual lib's.


To add a new lib, it's then as simple as (from the main repo) doing 
something like

- "git submodule add $URI"
- "./scripts/update-library-indexes.sh"
- "git commit -a -m $'added new module "modulename"\nupdated all 
module descriptions and index'"

- "git push"


All of the above allows the Main repo to be forked by a company (or 
individual) and have their own custom repo's added very easily.
It even allows libraries to be easily excluded or replaced, all using 
extremely well developed management tools.


On a side note, git submodules are stored in the main repository 
basically as URI that includes a commit reference.
That means it's easy to specify a specific library version to include 
in a specific branch/tag of the main repo.



As for the person that said "git isn't available for windows", sorry, 
that's bunkum.
There are many many development environments out there that ONLY run 
on windows that have git integration.
A quick google search for "git for windows" will show what you need 
to know there.


Finally, wrapping git to duplicate the effective API that's currently 
in use should be relatively trivial, resulting in virtually no code 
changes required to KICAD it's self.
Assuming the existing functionality is cleanly wrapped by an API, the 
only real change to KICAD would be the swapout of the module, and 
addition of an easy way to change the URI



I know I haven't covered everything in this email, but it should be a 
good outline for further discussion





___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp




___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Tiger12506]

+1 on this approach, to me this is the obvious way that it should be 
done. However, my opinion is not important, I'm not a contributor. :)


On 9/27/2017 8:37 PM, David Godfrey wrote:

Hi All

I have to agree with some of the other posters, why not use git as it 
was intended?


Specifically I think the following makes sense.

- Keep the multiple repositories, this helps when a company only wants 
to make certain sets of libs available to it's staff


- Have a master repository that includes all other repositories as 
submodules.


- Have branches that are matched to kicad versions. This allows 
footprint changes in a version safe way.


- Use standard "git clone" (initial download) and "git pull" (update) 
on the main repo which provides the entire set of available libs 
without actually downloading the content for all libs.


- When a specific lib is needed, do similar to what we do now, and use 
"git submodule update --init --recursive $submoduleName" to just pull 
that specific submodule


- Allow the "main" library repo URI to be altered. This enables a 
companies fork of the repo to be used.


- Allow the "main" library repo URI to be ANY valid git URI. That 
means a repo on a local fileserver rather than a http server can be 
used. Along with various security options etc.


- Add a fairly simple scripted tool that's run "on release" to 
retrieve a README.md (an possibly a descriptor/index file) from each 
actual library repo and update those within the "main" repo. This 
removes the need to have the current case where manual edits are 
required to keep the "main" repo in sync with what's available in the 
individual lib's.


To add a new lib, it's then as simple as (from the main repo) doing 
something like

- "git submodule add $URI"
- "./scripts/update-library-indexes.sh"
- "git commit -a -m $'added new module "modulename"\nupdated all 
module descriptions and index'"

- "git push"


All of the above allows the Main repo to be forked by a company (or 
individual) and have their own custom repo's added very easily.
It even allows libraries to be easily excluded or replaced, all using 
extremely well developed management tools.


On a side note, git submodules are stored in the main repository 
basically as URI that includes a commit reference.
That means it's easy to specify a specific library version to include 
in a specific branch/tag of the main repo.



As for the person that said "git isn't available for windows", sorry, 
that's bunkum.
There are many many development environments out there that ONLY run 
on windows that have git integration.
A quick google search for "git for windows" will show what you need to 
know there.


Finally, wrapping git to duplicate the effective API that's currently 
in use should be relatively trivial, resulting in virtually no code 
changes required to KICAD it's self.
Assuming the existing functionality is cleanly wrapped by an API, the 
only real change to KICAD would be the swapout of the module, and 
addition of an easy way to change the URI



I know I haven't covered everything in this email, but it should be a 
good outline for further discussion





___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[David Godfrey]

Hi All

I have to agree with some of the other posters, why not use git as it 
was intended?


Specifically I think the following makes sense.

- Keep the multiple repositories, this helps when a company only wants 
to make certain sets of libs available to it's staff


- Have a master repository that includes all other repositories as 
submodules.


- Have branches that are matched to kicad versions. This allows 
footprint changes in a version safe way.


- Use standard "git clone" (initial download) and "git pull" (update) on 
the main repo which provides the entire set of available libs without 
actually downloading the content for all libs.


- When a specific lib is needed, do similar to what we do now, and use 
"git submodule update --init --recursive $submoduleName" to just pull 
that specific submodule


- Allow the "main" library repo URI to be altered. This enables a 
companies fork of the repo to be used.


- Allow the "main" library repo URI to be ANY valid git URI. That means 
a repo on a local fileserver rather than a http server can be used. 
Along with various security options etc.


- Add a fairly simple scripted tool that's run "on release" to retrieve 
a README.md (an possibly a descriptor/index file) from each actual 
library repo and update those within the "main" repo. This removes the 
need to have the current case where manual edits are required to keep 
the "main" repo in sync with what's available in the individual lib's.


To add a new lib, it's then as simple as (from the main repo) doing 
something like

- "git submodule add $URI"
- "./scripts/update-library-indexes.sh"
- "git commit -a -m $'added new module "modulename"\nupdated all module 
descriptions and index'"

- "git push"


All of the above allows the Main repo to be forked by a company (or 
individual) and have their own custom repo's added very easily.
It even allows libraries to be easily excluded or replaced, all using 
extremely well developed management tools.


On a side note, git submodules are stored in the main repository 
basically as URI that includes a commit reference.
That means it's easy to specify a specific library version to include in 
a specific branch/tag of the main repo.



As for the person that said "git isn't available for windows", sorry, 
that's bunkum.
There are many many development environments out there that ONLY run on 
windows that have git integration.
A quick google search for "git for windows" will show what you need to 
know there.


Finally, wrapping git to duplicate the effective API that's currently in 
use should be relatively trivial, resulting in virtually no code changes 
required to KICAD it's self.
Assuming the existing functionality is cleanly wrapped by an API, the 
only real change to KICAD would be the swapout of the module, and 
addition of an easy way to change the URI



I know I haven't covered everything in this email, but it should be a 
good outline for further discussion



On 22/09/17 09:21, Oliver Walters wrote:

Hi all,

Ok, now that the website integration with the libraries is (pretty 
much) done, and the licensing issue seems to be sorted, there is one 
final puzzle piece to solve before I'm happy with the state of the 
libraries for a v5 release.


*Goal: *Merge all footprint library repositories into a single repo to 
solve the ongoing dramas of maintaining 100+ repos.


*Problem*: The *only* thing standing in the way of just doing this is 
that some users like the GitHub plugin and previous instruction is 
that this functionality cannot be removed.


The GitHub plugin functions by downloading a .zip file of a .pretty 
repo. If we merge all footprint libs into a single repo with multiple 
subdirectories, this will not work anymore (as GitHub dosen't allow 
you to download a .zip of a single subdirectory).


Merging the repos is the *right thing to do*. But how to proceed?

*Options:*
*
*
/a) Drop github plugin feature, replace with library-download tool/

I don't think it is a good idea to live-load library data from GitHub 
(a lot of other users agree too). It's slow, and a waste of bandwidth 
to re-download the libs all the time.


We drop support for loading libraries direct from GitHub. However, we 
add a tool for downloading libraries from GitHub and storing to disk. 
Users can update as they like. This can be integrated in KiCad and new 
users can run this tool when they first install KICad. This means that 
no libs need to distributed with the installer and users can update to 
latest libs whenever they want.


/b) Improve github plugin to allow subdirectory traversal/

This is difficult and will only result in the plugin being slower. 
There are two ways I can see to do this:


i. Use Git API - tools exist that use this functionality - 
https://github.com/KinoLien/gitzip
ii. Use subversion - GitHub actually provides subversion API - 
https://www.seanw.org/blog/download-git-repo-subdirectory/



*Subversion*

In either case, I think that

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Rene Pöschl]

On 23/09/17 20:51, Thomas Kindler wrote:

The other suggestions were quite surprising to me -- Github API, downloading
individual subdirectories as .zip, abusing subversion (gasp), gitslave (last
updated 2012), etc. would be a big hurdle and WTF for new KiCAD users.

Sorry for the late response. (I have been quite busy the last few weeks.)
I think there is a misunderstanding here. Using the github api to 
download individual repos as zip is not what has been suggested. It is 
the way it is currently implemented and the reason for this whole thread 
is to find a better solution to this.




___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Gaurav Juvekar]

Hi Thomas,

> So it's not worth the hassle IMHO as shallow clones don't allow to generate
> pull-requests without unshallowing them first.

Yes, it is possible without unshallowing as the server-side (github.com) fork 
into which you are pushing is a full clone, and the shallow history that you 
have locally has a common commit with the fork.

Simple git push your-fork pr-branch works in this case.

-- 
Regards,
Gaurav Juvekar

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Thomas Kindler]

On Sat, September 23, 2017 21:01, Simon Küppers wrote:
> [..]
>
> Maybe it would be a good way to implement two git repos, one for
> footprints and one for 3D models (and a third one for schematic symbols..).
> That is 3 repos to maintain instead of more than some dozens
> as it is right now. Then I would agree with you, I would personally be ok
> with downloading either all footprints or none and not be able to select
> the libraries to download.

I would prefer an all-or-nothing approach to symbols and footprints. They
take up very little space, and if all users have the same basic set of
libraries, it's great for support.

> And then the user can choose if 3D models
> should be pulled additionally.

Agreed. People with older computers may choose not to use them.

The default should be to clone everything though.


> However that is my opinion only. (What actually about stream compression
> for KiCad Libraries? Of course this would be a big change but could reduce
> just the disk and download size in general)
>

Download size will stay the same as git already uses compression.

.sweet and .pretty should stay uncompressed for easier diffing. But the
generated .step and .wrl files could be compressed (possibly not even
worth it, as it may make git compression worse).

> However, once the repos are pulled they need to be "tracked" somehow.
> Because I strongly disagree with everyone saying "Just do it as you do
> in software development, use the command line periodically to check for
> KiCad footprint repo changes". Just No.

As a user, I would expect the same behaviour as with commercial PCB tools:
Libraries are updated along with new software releases. That way the
libraries are guaranteed to work and are checked for compatibility.

For KiCAD, the 4.0.7 installer should check out the 4.0.7 branch initially.

For library-developers there could be a 4.0-next branch.

Nigthly versions could track master by default.


> A simple first idea I was having in mind for a long time is just a check
> on KiCad startup (by default, can be changed) what the status of the git
> remote is and _notify_ the user if something has changed and if they want
> to pull in the changes _to disk_.

KiCAD could check if the library branch matches the software version. If I
install KiCAD 4.1.0 (or so), it should offer to pull the updates and
checkout the new branch.

Extra care must be taken if the user made local modifications - we don't
want a merge or rebase here.


> Not altering any of the designs of course! This could be improved later
> to provide a more sophisticated GIT integration in KiCad (I get the warm
> fuzzy feeling when I write this :-))

A side-by-side graphical diff viewer would be nice!

-- 
Thomas Kindler

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Simon Küppers]

Your message got me thinking.

Maybe it would be a good way to implement two git repos, one for
footprints and one for 3D models (and a third one for schematic
symbols..). That is 3 repos to maintain instead of more than some dozens
as it is right now.
Then I would agree with you, I would personally be ok with downloading
either all footprints or none and not be able to select the libraries to
download. And then the user can choose if 3D models should be pulled
additionally. However that is my opinion only.
(What actually about stream compression for KiCad Libraries? Of course
this would be a big change but could reduce just the disk and download
size in general)

However, once the repos are pulled they need to be "tracked" somehow.
Because I strongly disagree with everyone saying "Just do it as you do
in software development, use the command line periodically to check for
KiCad footprint repo changes". Just No.
A simple first idea I was having in mind for a long time is just a check
on KiCad startup (by default, can be changed) what the status of the git
remote is and _notify_ the user if something has changed and if they
want to pull in the changes _to disk_. Not altering any of the designs
of course!
This could be improved later to provide a more sophisticated GIT
integration in KiCad (I get the warm fuzzy feeling when I write this :-))

> So why not use git as it was supposed to be used?
Strongly agree.

Best Regards


Am 23.09.2017 um 20:51 schrieb Thomas Kindler:
> Hi all!
> 
> After reading all the other messages, here are my two cents:
> 
> The KiCAD installer should just offer to do a full git clone of 
> kicad-footprints
> and kicad-libray (which will be split into 3D, templates, etc. for the 5.0
> release as I understand).
> 
> A full clone takes 3.8 GB. A shallow clone takes 3.6 GB. The .git-direcotry 
> size
> is 650 MB vs 486 MB.
> 
> So it's not worth the hassle IMHO as shallow clones don't allow to generate
> pull-requests without unshallowing them first.
> 
> BTW: 2 GB of that checkout is spent on the packages3d subdirectory. We could
> just support loading of .wrl.gz/.step.gz to save space.
> 
> In 2017, a few gig of harddisk space, and a < 1 gig download should be OK for
> everyone. All the other PCB tools, FPGA toolchains, Netflix, etc. are far 
> bigger.
> 
> The other suggestions were quite surprising to me -- Github API, downloading
> individual subdirectories as .zip, abusing subversion (gasp), gitslave (last
> updated 2012), etc. would be a big hurdle and WTF for new KiCAD users.
> 
> So why not use git as it was supposed to be used?
> 
> best regards,
> thomas
> 
> 
> On 22.09.2017 03:21, Oliver Walters wrote:
>> Hi all,
>>
>> Ok, now that the website integration with the libraries is (pretty much) 
>> done,
>> and the licensing issue seems to be sorted, there is one final puzzle piece 
>> to
>> solve before I'm happy with the state of the libraries for a v5 release.
>>
>> *Goal: *Merge all footprint library repositories into a single repo to solve 
>> the
>> ongoing dramas of maintaining 100+ repos.
>>
>> *Problem*: The *only* thing standing in the way of just doing this is that 
>> some
>> users like the GitHub plugin and previous instruction is that this 
>> functionality
>> cannot be removed.
>>
>> The GitHub plugin functions by downloading a .zip file of a .pretty repo. If 
>> we
>> merge all footprint libs into a single repo with multiple subdirectories, 
>> this
>> will not work anymore (as GitHub dosen't allow you to download a .zip of a
>> single subdirectory).
>>
>> Merging the repos is the *right thing to do*. But how to proceed?
>>
>> *Options:*
>> *
>> *
>> /a) Drop github plugin feature, replace with library-download tool/
>>
>> I don't think it is a good idea to live-load library data from GitHub (a lot 
>> of
>> other users agree too). It's slow, and a waste of bandwidth to re-download 
>> the
>> libs all the time. 
>>
>> We drop support for loading libraries direct from GitHub. However, we add a 
>> tool
>> for downloading libraries from GitHub and storing to disk. Users can update 
>> as
>> they like. This can be integrated in KiCad and new users can run this tool 
>> when
>> they first install KICad. This means that no libs need to distributed with 
>> the
>> installer and users can update to latest libs whenever they want.
>>
>> /b) Improve github plugin to allow subdirectory traversal/
>>
>> This is difficult and will only result in the plugin being slower. There are 
>> two
>> ways I can see to do this:
>>
>> i. Use Git API - tools exist that use this functionality
>> - https://github.com/KinoLien/gitzip
>> ii. Use subversion - GitHub actually provides subversion API
>> - https://www.seanw.org/blog/download-git-repo-subdirectory/
>>
>>
>> *Subversion*
>>
>> In either case, I think that using the subversion tool to partially download 
>> the
>> libraries would be a good approach (I assume quicker than using wget and the
>> GitHub API).
>>
>> 1. Does anyone hav

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Thomas Kindler]

Hi all!

After reading all the other messages, here are my two cents:

The KiCAD installer should just offer to do a full git clone of kicad-footprints
and kicad-libray (which will be split into 3D, templates, etc. for the 5.0
release as I understand).

A full clone takes 3.8 GB. A shallow clone takes 3.6 GB. The .git-direcotry size
is 650 MB vs 486 MB.

So it's not worth the hassle IMHO as shallow clones don't allow to generate
pull-requests without unshallowing them first.

BTW: 2 GB of that checkout is spent on the packages3d subdirectory. We could
just support loading of .wrl.gz/.step.gz to save space.

In 2017, a few gig of harddisk space, and a < 1 gig download should be OK for
everyone. All the other PCB tools, FPGA toolchains, Netflix, etc. are far 
bigger.

The other suggestions were quite surprising to me -- Github API, downloading
individual subdirectories as .zip, abusing subversion (gasp), gitslave (last
updated 2012), etc. would be a big hurdle and WTF for new KiCAD users.

So why not use git as it was supposed to be used?

best regards,
thomas


On 22.09.2017 03:21, Oliver Walters wrote:
> Hi all,
> 
> Ok, now that the website integration with the libraries is (pretty much) done,
> and the licensing issue seems to be sorted, there is one final puzzle piece to
> solve before I'm happy with the state of the libraries for a v5 release.
> 
> *Goal: *Merge all footprint library repositories into a single repo to solve 
> the
> ongoing dramas of maintaining 100+ repos.
> 
> *Problem*: The *only* thing standing in the way of just doing this is that 
> some
> users like the GitHub plugin and previous instruction is that this 
> functionality
> cannot be removed.
> 
> The GitHub plugin functions by downloading a .zip file of a .pretty repo. If 
> we
> merge all footprint libs into a single repo with multiple subdirectories, this
> will not work anymore (as GitHub dosen't allow you to download a .zip of a
> single subdirectory).
> 
> Merging the repos is the *right thing to do*. But how to proceed?
> 
> *Options:*
> *
> *
> /a) Drop github plugin feature, replace with library-download tool/
> 
> I don't think it is a good idea to live-load library data from GitHub (a lot 
> of
> other users agree too). It's slow, and a waste of bandwidth to re-download the
> libs all the time. 
> 
> We drop support for loading libraries direct from GitHub. However, we add a 
> tool
> for downloading libraries from GitHub and storing to disk. Users can update as
> they like. This can be integrated in KiCad and new users can run this tool 
> when
> they first install KICad. This means that no libs need to distributed with the
> installer and users can update to latest libs whenever they want.
> 
> /b) Improve github plugin to allow subdirectory traversal/
> 
> This is difficult and will only result in the plugin being slower. There are 
> two
> ways I can see to do this:
> 
> i. Use Git API - tools exist that use this functionality
> - https://github.com/KinoLien/gitzip
> ii. Use subversion - GitHub actually provides subversion API
> - https://www.seanw.org/blog/download-git-repo-subdirectory/
> 
> 
> *Subversion*
> 
> In either case, I think that using the subversion tool to partially download 
> the
> libraries would be a good approach (I assume quicker than using wget and the
> GitHub API).
> 
> 1. Does anyone have any experience using the C API for SVN? 
> 2. The Python bindings are pretty good - https://pypi.python.org/pypi/svn - 
> and
> much easier to use. However, can we make the library download tools dependent 
> on
> enabling the python plugin?
> 3. Is there a way to checkout a subversion remote to memory (to replicate the
> functionality of the current GitHub plugin)? If not, I'm not sure how to
> approach option b) above.
> 
> 
> Feedback appreciated. I think that it is very important especially for new 
> users
> that this is improved. The GitHub plugin is constantly causing headaches!
> 
> Cheers,
> Oliver
> 
> 
> ___
> Mailing list: https://launchpad.net/~kicad-developers
> Post to : kicad-developers@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
> 


___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Simon Küppers]

I think we are all still not sure where we are heading and what problems
we actually want and need to solve here. But the most critical problems
have already been outlined in this thread. See the first posts.


Am 23.09.2017 um 16:47 schrieb Carsten Schoenert:
> Am 23.09.2017 um 16:29 schrieb Simon Küppers:
>> Just to be clear, currently, the plugin does not use GIT. It uses a
>> proprietary GitHub Interface.
>>
>> I also checked out gitslave and similar stuff some time ago, but
>> internally, I was not really satisfied.
>> One point against something like this might be, that we actually make
>> the work of the maintainers even harder (?) by expecting them to use a
>> specific tool that is not even available on Windows (?). After all, the
>> idea is to make their life easier (and if there might pop out a standard
>> GIT Plugin at the end, nobody would complain :-))
>>
>> Looks like this is a really hard problem to solve using the standard
>> tools and protocols :-(
> 
> If it's not clear what problem is to solve than it's of course difficult
> to provide a generic solution for all OS.
> 
> Once again, what is the expectation on such a tool? I still can't see
> any accepted agreement what developers should programming? I would start
> here before thinking about which protocol can be used.
> 
> If the question is to be more flexible on remote available libraries and
> not only relay on the existing GitHub solution than please define first
> what a generic solution would or should be created. Otherwise just one
> more thing will evolving.
> 
> There is more than GitHub and GitLab outside. I can simply put a zip
> archive somewhere with some metadata inside anywhere for example. What
> need to be inside that metadata so a library manager can decide what the
> content is and where it needs to be locally stored.
> 
> That brings then the question, we need to be clear how and where to
> store such local data. And that of course for all operating systems.
> 
> 
> (PS: Please cut of that unneeded data if top posting is needed.)
> 

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Gaurav Juvekar]

Hi,

A small slightly tangent suggestion:
Can we somehow remove the need to mention every .pretty repo in the 
fp-lib-table?
Instead, can we just give a list of folders in which in turn will contain 
.pretty folders.
This will permit automatic addition of libs if more .pretty libs are created in 
the kicad official libs (just updating the folder with the official libs will 
make them available). The descriptions can be moved into a file within each 
.pretty folder itself.

A second benefit is that the user can manage multiple libs easily. For example, 
they could have the github repo with all .pretty libs in one folder, an 
organization library folder with shared libs, or a locally generated folder 
with local libs, and updating any of the folders will automatically update all 
available libs shown in kicad..

Pros:
VCS independent way of managing libs from multiple sources
Easier updating when new libs are created in the kicad official repos.

Cons:
Need to somehow handle if there are libs in two different folders with the same 
name.
Each .pretty folder would need it's own description file.


Also, a similar thing could be done for symbol-lib-table.

-- 
Regards,
Gaurav Juvekar

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Carsten Schoenert]

Am 23.09.2017 um 16:12 schrieb Kevin Cozens:
> On 2017-09-22 03:44 AM, Oliver Walters wrote:
>> that would certainly work but svn has the advantage of being able to 
>> pull selective directories from GitHub.
> 
> As an FYI, you can use git to deal with an SVN repo using git-svn package.

And this is even more slower than plain svn. ;-(

-- 
Regards
Carsten Schoenert

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Carsten Schoenert]

Am 23.09.2017 um 16:29 schrieb Simon Küppers:
> Just to be clear, currently, the plugin does not use GIT. It uses a
> proprietary GitHub Interface.
> 
> I also checked out gitslave and similar stuff some time ago, but
> internally, I was not really satisfied.
> One point against something like this might be, that we actually make
> the work of the maintainers even harder (?) by expecting them to use a
> specific tool that is not even available on Windows (?). After all, the
> idea is to make their life easier (and if there might pop out a standard
> GIT Plugin at the end, nobody would complain :-))
> 
> Looks like this is a really hard problem to solve using the standard
> tools and protocols :-(

If it's not clear what problem is to solve than it's of course difficult
to provide a generic solution for all OS.

Once again, what is the expectation on such a tool? I still can't see
any accepted agreement what developers should programming? I would start
here before thinking about which protocol can be used.

If the question is to be more flexible on remote available libraries and
not only relay on the existing GitHub solution than please define first
what a generic solution would or should be created. Otherwise just one
more thing will evolving.

There is more than GitHub and GitLab outside. I can simply put a zip
archive somewhere with some metadata inside anywhere for example. What
need to be inside that metadata so a library manager can decide what the
content is and where it needs to be locally stored.

That brings then the question, we need to be clear how and where to
store such local data. And that of course for all operating systems.


(PS: Please cut of that unneeded data if top posting is needed.)

-- 
Regards
Carsten Schoenert

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Simon Küppers]

Just to be clear, currently, the plugin does not use GIT. It uses a
proprietary GitHub Interface.

I also checked out gitslave and similar stuff some time ago, but
internally, I was not really satisfied.
One point against something like this might be, that we actually make
the work of the maintainers even harder (?) by expecting them to use a
specific tool that is not even available on Windows (?). After all, the
idea is to make their life easier (and if there might pop out a standard
GIT Plugin at the end, nobody would complain :-))

Looks like this is a really hard problem to solve using the standard
tools and protocols :-(


Am 23.09.2017 um 16:18 schrieb Jason Lund:
> My quarter-cent:
> 
> Echoing Teger12506's comment, there may be some ways to keep them as
> separate repos in GIT and keep them manageable. I've seen gitslave work
> fairly well in situations like this:
> 
> http://gitslave.sourceforge.net/
> 
> Continuing with using GIT sounds like the best option as opposed to
> moving to a different protocol, especially if there was a large effort
> put into a plugin for it on the KiCAD side; it is hugely supported and
> the user-base is probably larger at this point compared to others?
> 
> 
> On Fri, Sep 22, 2017 at 6:28 AM, Bastian Neumannn
> mailto:neumann.bast...@gmail.com>> wrote:
> 
> True, I see the problem with everything in one repo with branches.
> 
> Maybe it was not a good idea after all.
> 
> 2017-09-22 11:13 GMT+02:00 Simon Küppers  >:
> 
> Good point, actually.
> 
> 
> Am 22. September 2017 11:08:15 MESZ schrieb Miguel Angel Ajo
> Pelayo mailto:majop...@redhat.com>>:
> 
> I believe it's better if each library type has a single
> directory on the top of the libraries repo, in a single branch.
> 
> That would let you have branches like
> 
> master
> stable/4
> stable/5
> stable/6 later on,
> 
> and point the specific versions of kicad to such branches,
> in a way that an old version of kicad would not explode if
> new features appear in libraries in a later version.
> 
> 
> In that way, it's very easy to backport a change if the
> library is still backwards-compatible
> 
> git checkout stable/4
> git cherry-pick 
> git push
> 
> 
> 
> On Fri, Sep 22, 2017 at 11:05 AM, Miguel Angel Ajo Pelayo
> mailto:majop...@redhat.com>> wrote:
> 
> Please don't use branches for that.
> 
> Branches are to track separate development efforts or
> release cycles/stabilization.
> 
> Using branches, while it's possible was not the intent
> when git was designed.
> 
> If you do it that way, then you won't be able to use
> branches to track libraries in different stabilization
> phases, etc.
> 
> 
> 
> 
> On Fri, Sep 22, 2017 at 10:51 AM, Bastian Neumannn
>  > wrote:
> 
> I really like the idea of having one repo with all
> the .pretty folders in different branches. The
> master can have meta data about the branches.
> 
> That also gives the ability to manage library
> downloads as you can download the branch as a zip.
> 
> Using git for library management is ideally
> implemented as a plugin. With the ability to define
> own repositories as well. The library downloader can
> fetch the branch list and present a  selection to
> the user to fetch whatever the user want to fetch.
> 
> zip files of the branches can be mirrored on other
> servers as well for the people not having access to
> github.
> 
> Cheers,
> Basti
> 
> 2017-09-22 10:39 GMT+02:00 Simon Küppers
> mailto:simon.kuepp...@rub.de>>:
> 
> And by the way, this would be a feature that is
> completely new to the market (correct me if I'm
> wrong). Git integration into eda software.
> I only know of altium that has an svn interface
> and a proprietary vault. The features both of
> which could be (at some point) realized using git.
> Innovation is fun :-)
> 
> The idea of modifying a footprint from the
> standard lib, and generating a patch that could
> be directly send to the maintainers (maybe using
> 

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Jason Lund]

My quarter-cent:

Echoing Teger12506's comment, there may be some ways to keep them as
separate repos in GIT and keep them manageable. I've seen gitslave work
fairly well in situations like this:

http://gitslave.sourceforge.net/

Continuing with using GIT sounds like the best option as opposed to moving
to a different protocol, especially if there was a large effort put into a
plugin for it on the KiCAD side; it is hugely supported and the user-base
is probably larger at this point compared to others?


On Fri, Sep 22, 2017 at 6:28 AM, Bastian Neumannn  wrote:

> True, I see the problem with everything in one repo with branches.
>
> Maybe it was not a good idea after all.
>
> 2017-09-22 11:13 GMT+02:00 Simon Küppers :
>
>> Good point, actually.
>>
>>
>> Am 22. September 2017 11:08:15 MESZ schrieb Miguel Angel Ajo Pelayo <
>> majop...@redhat.com>:
>>>
>>> I believe it's better if each library type has a single directory on the
>>> top of the libraries repo, in a single branch.
>>>
>>> That would let you have branches like
>>>
>>> master
>>> stable/4
>>> stable/5
>>> stable/6 later on,
>>>
>>> and point the specific versions of kicad to such branches, in a way that
>>> an old version of kicad would not explode if new features appear in
>>> libraries in a later version.
>>>
>>>
>>> In that way, it's very easy to backport a change if the library is still
>>> backwards-compatible
>>>
>>> git checkout stable/4
>>> git cherry-pick 
>>> git push
>>>
>>>
>>>
>>> On Fri, Sep 22, 2017 at 11:05 AM, Miguel Angel Ajo Pelayo <
>>> majop...@redhat.com> wrote:
>>>
 Please don't use branches for that.

 Branches are to track separate development efforts or release
 cycles/stabilization.

 Using branches, while it's possible was not the intent when git was
 designed.

 If you do it that way, then you won't be able to use branches to track
 libraries in different stabilization phases, etc.




 On Fri, Sep 22, 2017 at 10:51 AM, Bastian Neumannn <
 neumann.bast...@gmail.com> wrote:

> I really like the idea of having one repo with all the .pretty folders
> in different branches. The master can have meta data about the branches.
>
> That also gives the ability to manage library downloads as you can
> download the branch as a zip.
>
> Using git for library management is ideally implemented as a plugin.
> With the ability to define own repositories as well. The library 
> downloader
> can fetch the branch list and present a  selection to the user to fetch
> whatever the user want to fetch.
>
> zip files of the branches can be mirrored on other servers as well for
> the people not having access to github.
>
> Cheers,
> Basti
>
> 2017-09-22 10:39 GMT+02:00 Simon Küppers :
>
>> And by the way, this would be a feature that is completely new to the
>> market (correct me if I'm wrong). Git integration into eda software.
>> I only know of altium that has an svn interface and a proprietary
>> vault. The features both of which could be (at some point) realized using
>> git.
>> Innovation is fun :-)
>>
>> The idea of modifying a footprint from the standard lib, and
>> generating a patch that could be directly send to the maintainers (maybe
>> using the very new library website) would make contributing very easy!
>>
>> Am 22. September 2017 10:13:49 MESZ schrieb Ingo Kletti <
>> ikle...@htwg-konstanz.de>:
>>
>>> Hi,
>>>
>>> Am 22.09.2017 um 09:44 schrieb Oliver Walters:
>>>
  [...] svn has the advantage of being able to
  pull selective directories from GitHub. You could present the user 
 with a
  list of which libraries they actually want to pull down

>>>
>>> So, just like JS (@tiger12506) I'm excited any time the git integration
>>> comes up for discussion.
>>>
>>> While I understand the initial focus on Github, it's just like Simon 
>>> stated:
>>>
>>>  Why not just ask the user for a working directory and pull the
  libraries there using actual git?
  This has the obvious advantage, that anyone can use this not only

>>> with > github but also with his or her own local repository..
>>>
>>> Without in-depth knowledge about git vs. git-plugin vs. svn:
>>>
>>> Will it be possible to use another repository besides Github?
>>>
>>> In our case, we require our students to maintain their project on a
>>> Gitlab server. This server also hosts the KiCad libraries that were
>>> created for internal purposes. ATM, it's not possible to just pull the
>>> latest version of the internal KiCad libraries from inside KiCad
>>>
>>> And it might not just be us. I think having a proper git integration
>>> could ease the library handling of many users.
>>>
>>> In the end, a proper git an

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Kevin Cozens]

On 2017-09-22 03:44 AM, Oliver Walters wrote:
that would certainly work but svn has the advantage of being able to 
pull selective directories from GitHub.


As an FYI, you can use git to deal with an SVN repo using git-svn package.

--
Cheers!

Kevin.

http://www.ve3syb.ca/   |"Nerds make the shiny things that distract
Owner of Elecraft K2 #2172  | the mouth-breathers, and that's why we're
| powerful!"
#include  | --Chris Hardwick

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Bastian Neumannn]

True, I see the problem with everything in one repo with branches.

Maybe it was not a good idea after all.

2017-09-22 11:13 GMT+02:00 Simon Küppers :

> Good point, actually.
>
>
> Am 22. September 2017 11:08:15 MESZ schrieb Miguel Angel Ajo Pelayo <
> majop...@redhat.com>:
>>
>> I believe it's better if each library type has a single directory on the
>> top of the libraries repo, in a single branch.
>>
>> That would let you have branches like
>>
>> master
>> stable/4
>> stable/5
>> stable/6 later on,
>>
>> and point the specific versions of kicad to such branches, in a way that
>> an old version of kicad would not explode if new features appear in
>> libraries in a later version.
>>
>>
>> In that way, it's very easy to backport a change if the library is still
>> backwards-compatible
>>
>> git checkout stable/4
>> git cherry-pick 
>> git push
>>
>>
>>
>> On Fri, Sep 22, 2017 at 11:05 AM, Miguel Angel Ajo Pelayo <
>> majop...@redhat.com> wrote:
>>
>>> Please don't use branches for that.
>>>
>>> Branches are to track separate development efforts or release
>>> cycles/stabilization.
>>>
>>> Using branches, while it's possible was not the intent when git was
>>> designed.
>>>
>>> If you do it that way, then you won't be able to use branches to track
>>> libraries in different stabilization phases, etc.
>>>
>>>
>>>
>>>
>>> On Fri, Sep 22, 2017 at 10:51 AM, Bastian Neumannn <
>>> neumann.bast...@gmail.com> wrote:
>>>
 I really like the idea of having one repo with all the .pretty folders
 in different branches. The master can have meta data about the branches.

 That also gives the ability to manage library downloads as you can
 download the branch as a zip.

 Using git for library management is ideally implemented as a plugin.
 With the ability to define own repositories as well. The library downloader
 can fetch the branch list and present a  selection to the user to fetch
 whatever the user want to fetch.

 zip files of the branches can be mirrored on other servers as well for
 the people not having access to github.

 Cheers,
 Basti

 2017-09-22 10:39 GMT+02:00 Simon Küppers :

> And by the way, this would be a feature that is completely new to the
> market (correct me if I'm wrong). Git integration into eda software.
> I only know of altium that has an svn interface and a proprietary
> vault. The features both of which could be (at some point) realized using
> git.
> Innovation is fun :-)
>
> The idea of modifying a footprint from the standard lib, and
> generating a patch that could be directly send to the maintainers (maybe
> using the very new library website) would make contributing very easy!
>
> Am 22. September 2017 10:13:49 MESZ schrieb Ingo Kletti <
> ikle...@htwg-konstanz.de>:
>
>> Hi,
>>
>> Am 22.09.2017 um 09:44 schrieb Oliver Walters:
>>
>>>  [...] svn has the advantage of being able to
>>>  pull selective directories from GitHub. You could present the user 
>>> with a
>>>  list of which libraries they actually want to pull down
>>>
>>
>> So, just like JS (@tiger12506) I'm excited any time the git integration
>> comes up for discussion.
>>
>> While I understand the initial focus on Github, it's just like Simon 
>> stated:
>>
>>  Why not just ask the user for a working directory and pull the
>>>  libraries there using actual git?
>>>  This has the obvious advantage, that anyone can use this not only
>>>
>> with > github but also with his or her own local repository..
>>
>> Without in-depth knowledge about git vs. git-plugin vs. svn:
>>
>> Will it be possible to use another repository besides Github?
>>
>> In our case, we require our students to maintain their project on a
>> Gitlab server. This server also hosts the KiCad libraries that were
>> created for internal purposes. ATM, it's not possible to just pull the
>> latest version of the internal KiCad libraries from inside KiCad
>>
>> And it might not just be us. I think having a proper git integration
>> could ease the library handling of many users.
>>
>> In the end, a proper git and/or svn integration would also open the
>> possibility to directly handle version management of KiCad projects from
>> inside KiCad.
>>
>> Regards,
>>
>> Ingo
>>
>> --
>>
>> Mailing list: https://launchpad.net/~kicad-developers
>> Post to : kicad-developers@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~kicad-developers
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
> ___
> Mailing list: https://launchpad.net/~kicad-developers
> Post to : kicad-developers@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~ki

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Simon Küppers]

Good point, actually.

Am 22. September 2017 11:08:15 MESZ schrieb Miguel Angel Ajo Pelayo 
:
>I believe it's better if each library type has a single directory on
>the
>top of the libraries repo, in a single branch.
>
>That would let you have branches like
>
>master
>stable/4
>stable/5
>stable/6 later on,
>
>and point the specific versions of kicad to such branches, in a way
>that an
>old version of kicad would not explode if new features appear in
>libraries
>in a later version.
>
>
>In that way, it's very easy to backport a change if the library is
>still
>backwards-compatible
>
>git checkout stable/4
>git cherry-pick 
>git push
>
>
>
>On Fri, Sep 22, 2017 at 11:05 AM, Miguel Angel Ajo Pelayo <
>majop...@redhat.com> wrote:
>
>> Please don't use branches for that.
>>
>> Branches are to track separate development efforts or release
>> cycles/stabilization.
>>
>> Using branches, while it's possible was not the intent when git was
>> designed.
>>
>> If you do it that way, then you won't be able to use branches to
>track
>> libraries in different stabilization phases, etc.
>>
>>
>>
>>
>> On Fri, Sep 22, 2017 at 10:51 AM, Bastian Neumannn <
>> neumann.bast...@gmail.com> wrote:
>>
>>> I really like the idea of having one repo with all the .pretty
>folders in
>>> different branches. The master can have meta data about the
>branches.
>>>
>>> That also gives the ability to manage library downloads as you can
>>> download the branch as a zip.
>>>
>>> Using git for library management is ideally implemented as a plugin.
>With
>>> the ability to define own repositories as well. The library
>downloader can
>>> fetch the branch list and present a  selection to the user to fetch
>>> whatever the user want to fetch.
>>>
>>> zip files of the branches can be mirrored on other servers as well
>for
>>> the people not having access to github.
>>>
>>> Cheers,
>>> Basti
>>>
>>> 2017-09-22 10:39 GMT+02:00 Simon Küppers :
>>>
 And by the way, this would be a feature that is completely new to
>the
 market (correct me if I'm wrong). Git integration into eda
>software.
 I only know of altium that has an svn interface and a proprietary
>vault.
 The features both of which could be (at some point) realized using
>git.
 Innovation is fun :-)

 The idea of modifying a footprint from the standard lib, and
>generating
 a patch that could be directly send to the maintainers (maybe using
>the
 very new library website) would make contributing very easy!

 Am 22. September 2017 10:13:49 MESZ schrieb Ingo Kletti <
 ikle...@htwg-konstanz.de>:

> Hi,
>
> Am 22.09.2017 um 09:44 schrieb Oliver Walters:
>
>>  [...] svn has the advantage of being able to
>>  pull selective directories from GitHub. You could present the
>user with a
>>  list of which libraries they actually want to pull down
>>
>
> So, just like JS (@tiger12506) I'm excited any time the git
>integration
> comes up for discussion.
>
> While I understand the initial focus on Github, it's just like
>Simon stated:
>
>  Why not just ask the user for a working directory and pull the
>>  libraries there using actual git?
>>  This has the obvious advantage, that anyone can use this not
>only
>>
> with > github but also with his or her own local repository..
>
> Without in-depth knowledge about git vs. git-plugin vs. svn:
>
> Will it be possible to use another repository besides Github?
>
> In our case, we require our students to maintain their project on
>a
> Gitlab server. This server also hosts the KiCad libraries that
>were
> created for internal purposes. ATM, it's not possible to just pull
>the
> latest version of the internal KiCad libraries from inside KiCad
>
> And it might not just be us. I think having a proper git
>integration
> could ease the library handling of many users.
>
> In the end, a proper git and/or svn integration would also open
>the
> possibility to directly handle version management of KiCad
>projects from
> inside KiCad.
>
> Regards,
>
> Ingo
>
> --
>
> Mailing list: https://launchpad.net/~kicad-developers
> Post to : kicad-developers@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>
>
 ___
 Mailing list: https://launchpad.net/~kicad-developers
 Post to : kicad-developers@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~kicad-developers
 More help   : https://help.launchpad.net/ListHelp


>>>
>>> ___
>>> Mailing list: https://launchpad.net/~kicad-developers
>>> Post to : kicad-developers@lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~kicad-developers
>>> More help   : htt

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Miguel Angel Ajo Pelayo]

I believe it's better if each library type has a single directory on the
top of the libraries repo, in a single branch.

That would let you have branches like

master
stable/4
stable/5
stable/6 later on,

and point the specific versions of kicad to such branches, in a way that an
old version of kicad would not explode if new features appear in libraries
in a later version.


In that way, it's very easy to backport a change if the library is still
backwards-compatible

git checkout stable/4
git cherry-pick 
git push



On Fri, Sep 22, 2017 at 11:05 AM, Miguel Angel Ajo Pelayo <
majop...@redhat.com> wrote:

> Please don't use branches for that.
>
> Branches are to track separate development efforts or release
> cycles/stabilization.
>
> Using branches, while it's possible was not the intent when git was
> designed.
>
> If you do it that way, then you won't be able to use branches to track
> libraries in different stabilization phases, etc.
>
>
>
>
> On Fri, Sep 22, 2017 at 10:51 AM, Bastian Neumannn <
> neumann.bast...@gmail.com> wrote:
>
>> I really like the idea of having one repo with all the .pretty folders in
>> different branches. The master can have meta data about the branches.
>>
>> That also gives the ability to manage library downloads as you can
>> download the branch as a zip.
>>
>> Using git for library management is ideally implemented as a plugin. With
>> the ability to define own repositories as well. The library downloader can
>> fetch the branch list and present a  selection to the user to fetch
>> whatever the user want to fetch.
>>
>> zip files of the branches can be mirrored on other servers as well for
>> the people not having access to github.
>>
>> Cheers,
>> Basti
>>
>> 2017-09-22 10:39 GMT+02:00 Simon Küppers :
>>
>>> And by the way, this would be a feature that is completely new to the
>>> market (correct me if I'm wrong). Git integration into eda software.
>>> I only know of altium that has an svn interface and a proprietary vault.
>>> The features both of which could be (at some point) realized using git.
>>> Innovation is fun :-)
>>>
>>> The idea of modifying a footprint from the standard lib, and generating
>>> a patch that could be directly send to the maintainers (maybe using the
>>> very new library website) would make contributing very easy!
>>>
>>> Am 22. September 2017 10:13:49 MESZ schrieb Ingo Kletti <
>>> ikle...@htwg-konstanz.de>:
>>>
 Hi,

 Am 22.09.2017 um 09:44 schrieb Oliver Walters:

>  [...] svn has the advantage of being able to
>  pull selective directories from GitHub. You could present the user with a
>  list of which libraries they actually want to pull down
>

 So, just like JS (@tiger12506) I'm excited any time the git integration
 comes up for discussion.

 While I understand the initial focus on Github, it's just like Simon 
 stated:

  Why not just ask the user for a working directory and pull the
>  libraries there using actual git?
>  This has the obvious advantage, that anyone can use this not only
>
 with > github but also with his or her own local repository..

 Without in-depth knowledge about git vs. git-plugin vs. svn:

 Will it be possible to use another repository besides Github?

 In our case, we require our students to maintain their project on a
 Gitlab server. This server also hosts the KiCad libraries that were
 created for internal purposes. ATM, it's not possible to just pull the
 latest version of the internal KiCad libraries from inside KiCad

 And it might not just be us. I think having a proper git integration
 could ease the library handling of many users.

 In the end, a proper git and/or svn integration would also open the
 possibility to directly handle version management of KiCad projects from
 inside KiCad.

 Regards,

 Ingo

 --

 Mailing list: https://launchpad.net/~kicad-developers
 Post to : kicad-developers@lists.launchpad.net
 Unsubscribe : https://launchpad.net/~kicad-developers
 More help   : https://help.launchpad.net/ListHelp


>>> ___
>>> Mailing list: https://launchpad.net/~kicad-developers
>>> Post to : kicad-developers@lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~kicad-developers
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>> ___
>> Mailing list: https://launchpad.net/~kicad-developers
>> Post to : kicad-developers@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~kicad-developers
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
Mor

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Miguel Angel Ajo Pelayo]

Please don't use branches for that.

Branches are to track separate development efforts or release
cycles/stabilization.

Using branches, while it's possible was not the intent when git was
designed.

If you do it that way, then you won't be able to use branches to track
libraries in different stabilization phases, etc.




On Fri, Sep 22, 2017 at 10:51 AM, Bastian Neumannn <
neumann.bast...@gmail.com> wrote:

> I really like the idea of having one repo with all the .pretty folders in
> different branches. The master can have meta data about the branches.
>
> That also gives the ability to manage library downloads as you can
> download the branch as a zip.
>
> Using git for library management is ideally implemented as a plugin. With
> the ability to define own repositories as well. The library downloader can
> fetch the branch list and present a  selection to the user to fetch
> whatever the user want to fetch.
>
> zip files of the branches can be mirrored on other servers as well for the
> people not having access to github.
>
> Cheers,
> Basti
>
> 2017-09-22 10:39 GMT+02:00 Simon Küppers :
>
>> And by the way, this would be a feature that is completely new to the
>> market (correct me if I'm wrong). Git integration into eda software.
>> I only know of altium that has an svn interface and a proprietary vault.
>> The features both of which could be (at some point) realized using git.
>> Innovation is fun :-)
>>
>> The idea of modifying a footprint from the standard lib, and generating a
>> patch that could be directly send to the maintainers (maybe using the very
>> new library website) would make contributing very easy!
>>
>> Am 22. September 2017 10:13:49 MESZ schrieb Ingo Kletti <
>> ikle...@htwg-konstanz.de>:
>>
>>> Hi,
>>>
>>> Am 22.09.2017 um 09:44 schrieb Oliver Walters:
>>>
  [...] svn has the advantage of being able to
  pull selective directories from GitHub. You could present the user with a
  list of which libraries they actually want to pull down

>>>
>>> So, just like JS (@tiger12506) I'm excited any time the git integration
>>> comes up for discussion.
>>>
>>> While I understand the initial focus on Github, it's just like Simon stated:
>>>
>>>  Why not just ask the user for a working directory and pull the
  libraries there using actual git?
  This has the obvious advantage, that anyone can use this not only

>>> with > github but also with his or her own local repository..
>>>
>>> Without in-depth knowledge about git vs. git-plugin vs. svn:
>>>
>>> Will it be possible to use another repository besides Github?
>>>
>>> In our case, we require our students to maintain their project on a
>>> Gitlab server. This server also hosts the KiCad libraries that were
>>> created for internal purposes. ATM, it's not possible to just pull the
>>> latest version of the internal KiCad libraries from inside KiCad
>>>
>>> And it might not just be us. I think having a proper git integration
>>> could ease the library handling of many users.
>>>
>>> In the end, a proper git and/or svn integration would also open the
>>> possibility to directly handle version management of KiCad projects from
>>> inside KiCad.
>>>
>>> Regards,
>>>
>>> Ingo
>>>
>>> --
>>>
>>> Mailing list: https://launchpad.net/~kicad-developers
>>> Post to : kicad-developers@lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~kicad-developers
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>> ___
>> Mailing list: https://launchpad.net/~kicad-developers
>> Post to : kicad-developers@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~kicad-developers
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
> ___
> Mailing list: https://launchpad.net/~kicad-developers
> Post to : kicad-developers@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>
>
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Simon Küppers]

That actually sounds like a nice idea. Also the development of a single library 
would be local to the branch itself. Which could come in handy.. 
Then master could also hold Metadata (in a text file, but machine readable) of 
e.g. The address of the maintainer (email?) where changes are to be submitted. 

Am 22. September 2017 10:51:25 MESZ schrieb Bastian Neumannn 
:
>I really like the idea of having one repo with all the .pretty folders
>in
>different branches. The master can have meta data about the branches.
>
>That also gives the ability to manage library downloads as you can
>download
>the branch as a zip.
>
>Using git for library management is ideally implemented as a plugin.
>With
>the ability to define own repositories as well. The library downloader
>can
>fetch the branch list and present a  selection to the user to fetch
>whatever the user want to fetch.
>
>zip files of the branches can be mirrored on other servers as well for
>the
>people not having access to github.
>
>Cheers,
>Basti
>
>2017-09-22 10:39 GMT+02:00 Simon Küppers :
>
>> And by the way, this would be a feature that is completely new to the
>> market (correct me if I'm wrong). Git integration into eda software.
>> I only know of altium that has an svn interface and a proprietary
>vault.
>> The features both of which could be (at some point) realized using
>git.
>> Innovation is fun :-)
>>
>> The idea of modifying a footprint from the standard lib, and
>generating a
>> patch that could be directly send to the maintainers (maybe using the
>very
>> new library website) would make contributing very easy!
>>
>> Am 22. September 2017 10:13:49 MESZ schrieb Ingo Kletti <
>> ikle...@htwg-konstanz.de>:
>>
>>> Hi,
>>>
>>> Am 22.09.2017 um 09:44 schrieb Oliver Walters:
>>>
  [...] svn has the advantage of being able to
  pull selective directories from GitHub. You could present the user
>with a
  list of which libraries they actually want to pull down

>>>
>>> So, just like JS (@tiger12506) I'm excited any time the git
>integration
>>> comes up for discussion.
>>>
>>> While I understand the initial focus on Github, it's just like Simon
>stated:
>>>
>>>  Why not just ask the user for a working directory and pull the
  libraries there using actual git?
  This has the obvious advantage, that anyone can use this not only

>>> with > github but also with his or her own local repository..
>>>
>>> Without in-depth knowledge about git vs. git-plugin vs. svn:
>>>
>>> Will it be possible to use another repository besides Github?
>>>
>>> In our case, we require our students to maintain their project on a
>>> Gitlab server. This server also hosts the KiCad libraries that were
>>> created for internal purposes. ATM, it's not possible to just pull
>the
>>> latest version of the internal KiCad libraries from inside KiCad
>>>
>>> And it might not just be us. I think having a proper git integration
>>> could ease the library handling of many users.
>>>
>>> In the end, a proper git and/or svn integration would also open the
>>> possibility to directly handle version management of KiCad projects
>from
>>> inside KiCad.
>>>
>>> Regards,
>>>
>>> Ingo
>>>
>>> --
>>>
>>> Mailing list: https://launchpad.net/~kicad-developers
>>> Post to : kicad-developers@lists.launchpad.net
>>> Unsubscribe : https://launchpad.net/~kicad-developers
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>> ___
>> Mailing list: https://launchpad.net/~kicad-developers
>> Post to : kicad-developers@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~kicad-developers
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Bastian Neumannn]

I really like the idea of having one repo with all the .pretty folders in
different branches. The master can have meta data about the branches.

That also gives the ability to manage library downloads as you can download
the branch as a zip.

Using git for library management is ideally implemented as a plugin. With
the ability to define own repositories as well. The library downloader can
fetch the branch list and present a  selection to the user to fetch
whatever the user want to fetch.

zip files of the branches can be mirrored on other servers as well for the
people not having access to github.

Cheers,
Basti

2017-09-22 10:39 GMT+02:00 Simon Küppers :

> And by the way, this would be a feature that is completely new to the
> market (correct me if I'm wrong). Git integration into eda software.
> I only know of altium that has an svn interface and a proprietary vault.
> The features both of which could be (at some point) realized using git.
> Innovation is fun :-)
>
> The idea of modifying a footprint from the standard lib, and generating a
> patch that could be directly send to the maintainers (maybe using the very
> new library website) would make contributing very easy!
>
> Am 22. September 2017 10:13:49 MESZ schrieb Ingo Kletti <
> ikle...@htwg-konstanz.de>:
>
>> Hi,
>>
>> Am 22.09.2017 um 09:44 schrieb Oliver Walters:
>>
>>>  [...] svn has the advantage of being able to
>>>  pull selective directories from GitHub. You could present the user with a
>>>  list of which libraries they actually want to pull down
>>>
>>
>> So, just like JS (@tiger12506) I'm excited any time the git integration
>> comes up for discussion.
>>
>> While I understand the initial focus on Github, it's just like Simon stated:
>>
>>  Why not just ask the user for a working directory and pull the
>>>  libraries there using actual git?
>>>  This has the obvious advantage, that anyone can use this not only
>>>
>> with > github but also with his or her own local repository..
>>
>> Without in-depth knowledge about git vs. git-plugin vs. svn:
>>
>> Will it be possible to use another repository besides Github?
>>
>> In our case, we require our students to maintain their project on a
>> Gitlab server. This server also hosts the KiCad libraries that were
>> created for internal purposes. ATM, it's not possible to just pull the
>> latest version of the internal KiCad libraries from inside KiCad
>>
>> And it might not just be us. I think having a proper git integration
>> could ease the library handling of many users.
>>
>> In the end, a proper git and/or svn integration would also open the
>> possibility to directly handle version management of KiCad projects from
>> inside KiCad.
>>
>> Regards,
>>
>> Ingo
>>
>> --
>>
>> Mailing list: https://launchpad.net/~kicad-developers
>> Post to : kicad-developers@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~kicad-developers
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
> ___
> Mailing list: https://launchpad.net/~kicad-developers
> Post to : kicad-developers@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>
>
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Simon Küppers]

And by the way, this would be a feature that is completely new to the market 
(correct me if I'm wrong). Git integration into eda software. 
I only know of altium that has an svn interface and a proprietary vault. The 
features both of which could be (at some point) realized using git. 
Innovation is fun :-) 

The idea of modifying a footprint from the standard lib, and generating a patch 
that could be directly send to the maintainers (maybe using the very new 
library website) would make contributing very easy!

Am 22. September 2017 10:13:49 MESZ schrieb Ingo Kletti 
:
>Hi,
>
>Am 22.09.2017 um 09:44 schrieb Oliver Walters:
>> [...] svn has the advantage of being able to
>> pull selective directories from GitHub. You could present the user
>with a
>> list of which libraries they actually want to pull down
>
>So, just like JS (@tiger12506) I'm excited any time the git integration
>
>comes up for discussion.
>
>While I understand the initial focus on Github, it's just like Simon
>stated:
>
> > Why not just ask the user for a working directory and pull the
> > libraries there using actual git?
> > This has the obvious advantage, that anyone can use this not only 
>with > github but also with his or her own local repository..
>
>Without in-depth knowledge about git vs. git-plugin vs. svn:
>
>Will it be possible to use another repository besides Github?
>
>In our case, we require our students to maintain their project on a 
>Gitlab server. This server also hosts the KiCad libraries that were 
>created for internal purposes. ATM, it's not possible to just pull the 
>latest version of the internal KiCad libraries from inside KiCad
>
>And it might not just be us. I think having a proper git integration 
>could ease the library handling of many users.
>
>In the end, a proper git and/or svn integration would also open the 
>possibility to directly handle version management of KiCad projects
>from 
>inside KiCad.
>
>Regards,
>
>Ingo
>
>___
>Mailing list: https://launchpad.net/~kicad-developers
>Post to : kicad-developers@lists.launchpad.net
>Unsubscribe : https://launchpad.net/~kicad-developers
>More help   : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Simon Küppers]

As said in this thread title, the plugin-idea is exactly what people are 
talking about here. There is a plugin interface within kicad for such stuff. 

Am 22. September 2017 10:32:21 MESZ schrieb Seppe Stas :
>Honestly, I think KiCad should not manage library downloads by itself.
>If
>this is a feature people want to use, it should be developed as a
>plugin.
>This way multiple plugins can exist that support different use-cases,
>like
>downloading plugins from git, a file server or SVN.
>
>What are the "ongoing dramas of maintaining 100+ repos"?  Maybe some of
>the
>issues could be solved by creating a big repo that has the other repos
>as
>git submodules?
>
>Having the footprint libraries in different repos gives a lot of
>flexibility. E.g it allows us (Productize) to pick some of the
>community footprints and add them to our big Productize library repo as
>a
>submodule. Having all the footprints in a single library would cause
>conflicts with footprints we made ourselves.
>
>For my personal use I have a couple of forks of the community footprint
>libraries containing my own footprints. All the other footprints I just
>checked out directly from KiCad's github or installed using the
>KiCad-extras package. This makes it very easy to send small PRs when I
>want
>to share footprints. Having a single footprint library would make
>things
>way more heavy and inefficient (KiCad would have to load a bunch of
>footprints I don't use. Git would have history on a things I don't need
>history for or don't need at all. My filesystem would contain a bunch
>of
>files I don't need).
>
>Seppe
>
>2017-09-22 10:13 GMT+02:00 Ingo Kletti :
>
>> Hi,
>>
>> Am 22.09.2017 um 09:44 schrieb Oliver Walters:
>>
>>> [...] svn has the advantage of being able to
>>> pull selective directories from GitHub. You could present the user
>with a
>>> list of which libraries they actually want to pull down
>>>
>>
>> So, just like JS (@tiger12506) I'm excited any time the git
>integration
>> comes up for discussion.
>>
>> While I understand the initial focus on Github, it's just like Simon
>> stated:
>>
>> > Why not just ask the user for a working directory and pull the
>> > libraries there using actual git?
>> > This has the obvious advantage, that anyone can use this not only
>with >
>> github but also with his or her own local repository..
>>
>> Without in-depth knowledge about git vs. git-plugin vs. svn:
>>
>> Will it be possible to use another repository besides Github?
>>
>> In our case, we require our students to maintain their project on a
>Gitlab
>> server. This server also hosts the KiCad libraries that were created
>for
>> internal purposes. ATM, it's not possible to just pull the latest
>version
>> of the internal KiCad libraries from inside KiCad
>>
>> And it might not just be us. I think having a proper git integration
>could
>> ease the library handling of many users.
>>
>> In the end, a proper git and/or svn integration would also open the
>> possibility to directly handle version management of KiCad projects
>from
>> inside KiCad.
>>
>> Regards,
>>
>> Ingo
>>
>>
>> ___
>> Mailing list: https://launchpad.net/~kicad-developers
>> Post to : kicad-developers@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~kicad-developers
>> More help   : https://help.launchpad.net/ListHelp
>>
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Seppe Stas]

Honestly, I think KiCad should not manage library downloads by itself. If
this is a feature people want to use, it should be developed as a plugin.
This way multiple plugins can exist that support different use-cases, like
downloading plugins from git, a file server or SVN.

What are the "ongoing dramas of maintaining 100+ repos"?  Maybe some of the
issues could be solved by creating a big repo that has the other repos as
git submodules?

Having the footprint libraries in different repos gives a lot of
flexibility. E.g it allows us (Productize) to pick some of the
community footprints and add them to our big Productize library repo as a
submodule. Having all the footprints in a single library would cause
conflicts with footprints we made ourselves.

For my personal use I have a couple of forks of the community footprint
libraries containing my own footprints. All the other footprints I just
checked out directly from KiCad's github or installed using the
KiCad-extras package. This makes it very easy to send small PRs when I want
to share footprints. Having a single footprint library would make things
way more heavy and inefficient (KiCad would have to load a bunch of
footprints I don't use. Git would have history on a things I don't need
history for or don't need at all. My filesystem would contain a bunch of
files I don't need).

Seppe

2017-09-22 10:13 GMT+02:00 Ingo Kletti :

> Hi,
>
> Am 22.09.2017 um 09:44 schrieb Oliver Walters:
>
>> [...] svn has the advantage of being able to
>> pull selective directories from GitHub. You could present the user with a
>> list of which libraries they actually want to pull down
>>
>
> So, just like JS (@tiger12506) I'm excited any time the git integration
> comes up for discussion.
>
> While I understand the initial focus on Github, it's just like Simon
> stated:
>
> > Why not just ask the user for a working directory and pull the
> > libraries there using actual git?
> > This has the obvious advantage, that anyone can use this not only with >
> github but also with his or her own local repository..
>
> Without in-depth knowledge about git vs. git-plugin vs. svn:
>
> Will it be possible to use another repository besides Github?
>
> In our case, we require our students to maintain their project on a Gitlab
> server. This server also hosts the KiCad libraries that were created for
> internal purposes. ATM, it's not possible to just pull the latest version
> of the internal KiCad libraries from inside KiCad
>
> And it might not just be us. I think having a proper git integration could
> ease the library handling of many users.
>
> In the end, a proper git and/or svn integration would also open the
> possibility to directly handle version management of KiCad projects from
> inside KiCad.
>
> Regards,
>
> Ingo
>
>
> ___
> Mailing list: https://launchpad.net/~kicad-developers
> Post to : kicad-developers@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Ingo Kletti]

Hi,

Am 22.09.2017 um 09:44 schrieb Oliver Walters:

[...] svn has the advantage of being able to
pull selective directories from GitHub. You could present the user with a
list of which libraries they actually want to pull down


So, just like JS (@tiger12506) I'm excited any time the git integration 
comes up for discussion.


While I understand the initial focus on Github, it's just like Simon stated:

> Why not just ask the user for a working directory and pull the
> libraries there using actual git?
> This has the obvious advantage, that anyone can use this not only 
with > github but also with his or her own local repository..


Without in-depth knowledge about git vs. git-plugin vs. svn:

Will it be possible to use another repository besides Github?

In our case, we require our students to maintain their project on a 
Gitlab server. This server also hosts the KiCad libraries that were 
created for internal purposes. ATM, it's not possible to just pull the 
latest version of the internal KiCad libraries from inside KiCad


And it might not just be us. I think having a proper git integration 
could ease the library handling of many users.


In the end, a proper git and/or svn integration would also open the 
possibility to directly handle version management of KiCad projects from 
inside KiCad.


Regards,

Ingo

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Miguel Angel Ajo Pelayo]

My two cents (take with a grain of salt since I can't commit to it):
https://libgit2.github.com/docs/guides/101-samples/

Having the plugin use the libgit2 library would be great, even people could
get incremental updates via git, or even send patches/pull requests if they
wanted...



On Fri, Sep 22, 2017 at 10:03 AM, Maciej Sumiński 
wrote:

> Hi Oliver,
>
> IMHO, the library-download tool is the best choice here. I agree the
> live download is not a good idea in many cases, but I imagine there are
> people taking advantage of that (e.g. a centralized footprint repository
> in a small company).
>
> I think the main problem now, is we are relying too much on the Github
> API, instead of using e.g. libgit2 [1]. Such library is not only more
> flexible but also frees us from the Github. Even though it might
> impossible to download a .zip file of a whole repository, there should
> be an option to select single files to be downloaded or quickly list
> directories. Apart from that, updates will be much faster, as you do not
> need to download everything, but only deltas.
>
> I am sure the downloader could be also implemented in Python and
> integrated with pcbnew. IIRC all nightly builds are build with Python
> scripting emabled, and I am not aware of any problems related to
> compiling such version manually.
>
> Regards,
> Orson
>
> 1. https://github.com/libgit2/libgit2
>
> On 09/22/2017 03:21 AM, Oliver Walters wrote:
> > Hi all,
> >
> > Ok, now that the website integration with the libraries is (pretty much)
> > done, and the licensing issue seems to be sorted, there is one final
> puzzle
> > piece to solve before I'm happy with the state of the libraries for a v5
> > release.
> >
> > *Goal: *Merge all footprint library repositories into a single repo to
> > solve the ongoing dramas of maintaining 100+ repos.
> >
> > *Problem*: The *only* thing standing in the way of just doing this is
> that
> > some users like the GitHub plugin and previous instruction is that this
> > functionality cannot be removed.
> >
> > The GitHub plugin functions by downloading a .zip file of a .pretty repo.
> > If we merge all footprint libs into a single repo with multiple
> > subdirectories, this will not work anymore (as GitHub dosen't allow you
> to
> > download a .zip of a single subdirectory).
> >
> > Merging the repos is the *right thing to do*. But how to proceed?
> >
> > *Options:*
> >
> > *a) Drop github plugin feature, replace with library-download tool*
> >
> > I don't think it is a good idea to live-load library data from GitHub (a
> > lot of other users agree too). It's slow, and a waste of bandwidth to
> > re-download the libs all the time.
> >
> > We drop support for loading libraries direct from GitHub. However, we
> add a
> > tool for downloading libraries from GitHub and storing to disk. Users can
> > update as they like. This can be integrated in KiCad and new users can
> run
> > this tool when they first install KICad. This means that no libs need to
> > distributed with the installer and users can update to latest libs
> whenever
> > they want.
> >
> > *b) Improve github plugin to allow subdirectory traversal*
> >
> > This is difficult and will only result in the plugin being slower. There
> > are two ways I can see to do this:
> >
> > i. Use Git API - tools exist that use this functionality -
> > https://github.com/KinoLien/gitzip
> > ii. Use subversion - GitHub actually provides subversion API -
> > https://www.seanw.org/blog/download-git-repo-subdirectory/
> >
> >
> > *Subversion*
> >
> > In either case, I think that using the subversion tool to partially
> > download the libraries would be a good approach (I assume quicker than
> > using wget and the GitHub API).
> >
> > 1. Does anyone have any experience using the C API for SVN?
> > 2. The Python bindings are pretty good - https://pypi.python.org/pypi/
> svn -
> > and much easier to use. However, can we make the library download tools
> > dependent on enabling the python plugin?
> > 3. Is there a way to checkout a subversion remote to memory (to replicate
> > the functionality of the current GitHub plugin)? If not, I'm not sure how
> > to approach option b) above.
> >
> >
> > Feedback appreciated. I think that it is very important especially for
> new
> > users that this is improved. The GitHub plugin is constantly causing
> > headaches!
> >
> > Cheers,
> > Oliver
> >
> >
> >
> > ___
> > Mailing list: https://launchpad.net/~kicad-developers
> > Post to : kicad-developers@lists.launchpad.net
> > Unsubscribe : https://launchpad.net/~kicad-developers
> > More help   : https://help.launchpad.net/ListHelp
> >
>
>
>
> ___
> Mailing list: https://launchpad.net/~kicad-developers
> Post to : kicad-developers@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>
>
__

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Maciej Sumiński]

Hi Oliver,

IMHO, the library-download tool is the best choice here. I agree the
live download is not a good idea in many cases, but I imagine there are
people taking advantage of that (e.g. a centralized footprint repository
in a small company).

I think the main problem now, is we are relying too much on the Github
API, instead of using e.g. libgit2 [1]. Such library is not only more
flexible but also frees us from the Github. Even though it might
impossible to download a .zip file of a whole repository, there should
be an option to select single files to be downloaded or quickly list
directories. Apart from that, updates will be much faster, as you do not
need to download everything, but only deltas.

I am sure the downloader could be also implemented in Python and
integrated with pcbnew. IIRC all nightly builds are build with Python
scripting emabled, and I am not aware of any problems related to
compiling such version manually.

Regards,
Orson

1. https://github.com/libgit2/libgit2

On 09/22/2017 03:21 AM, Oliver Walters wrote:
> Hi all,
> 
> Ok, now that the website integration with the libraries is (pretty much)
> done, and the licensing issue seems to be sorted, there is one final puzzle
> piece to solve before I'm happy with the state of the libraries for a v5
> release.
> 
> *Goal: *Merge all footprint library repositories into a single repo to
> solve the ongoing dramas of maintaining 100+ repos.
> 
> *Problem*: The *only* thing standing in the way of just doing this is that
> some users like the GitHub plugin and previous instruction is that this
> functionality cannot be removed.
> 
> The GitHub plugin functions by downloading a .zip file of a .pretty repo.
> If we merge all footprint libs into a single repo with multiple
> subdirectories, this will not work anymore (as GitHub dosen't allow you to
> download a .zip of a single subdirectory).
> 
> Merging the repos is the *right thing to do*. But how to proceed?
> 
> *Options:*
> 
> *a) Drop github plugin feature, replace with library-download tool*
> 
> I don't think it is a good idea to live-load library data from GitHub (a
> lot of other users agree too). It's slow, and a waste of bandwidth to
> re-download the libs all the time.
> 
> We drop support for loading libraries direct from GitHub. However, we add a
> tool for downloading libraries from GitHub and storing to disk. Users can
> update as they like. This can be integrated in KiCad and new users can run
> this tool when they first install KICad. This means that no libs need to
> distributed with the installer and users can update to latest libs whenever
> they want.
> 
> *b) Improve github plugin to allow subdirectory traversal*
> 
> This is difficult and will only result in the plugin being slower. There
> are two ways I can see to do this:
> 
> i. Use Git API - tools exist that use this functionality -
> https://github.com/KinoLien/gitzip
> ii. Use subversion - GitHub actually provides subversion API -
> https://www.seanw.org/blog/download-git-repo-subdirectory/
> 
> 
> *Subversion*
> 
> In either case, I think that using the subversion tool to partially
> download the libraries would be a good approach (I assume quicker than
> using wget and the GitHub API).
> 
> 1. Does anyone have any experience using the C API for SVN?
> 2. The Python bindings are pretty good - https://pypi.python.org/pypi/svn -
> and much easier to use. However, can we make the library download tools
> dependent on enabling the python plugin?
> 3. Is there a way to checkout a subversion remote to memory (to replicate
> the functionality of the current GitHub plugin)? If not, I'm not sure how
> to approach option b) above.
> 
> 
> Feedback appreciated. I think that it is very important especially for new
> users that this is improved. The GitHub plugin is constantly causing
> headaches!
> 
> Cheers,
> Oliver
> 
> 
> 
> ___
> Mailing list: https://launchpad.net/~kicad-developers
> Post to : kicad-developers@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
> 




signature.asc
Description: OpenPGP digital signature
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Oliver Walters]

At this point, you can download a .zip or a .tar.gz from github of the
libraries. The new library info will be put on the website too, so easiest
option is just let the users download libraries direct from the KiCad
website!

If this is all that the git plugin is offering, I say that the middle-man
should be cut out :)

On Fri, Sep 22, 2017 at 5:44 PM, Oliver Walters <
oliver.henry.walt...@gmail.com> wrote:

> Simon, that would certainly work but svn has the advantage of being able
> to pull selective directories from GitHub. You could present the user with
> a list of which libraries they actually want to pull down
>
> Disadvantage is that (as you say) non github repos can't be used. Perhaps
> that's fine.
>
> On Fri, Sep 22, 2017 at 5:38 PM, Simon Küppers 
> wrote:
>
>> Sorry if I don't get it.. Why not just ask the user for a working
>> directory and pull the libraries there using actual git?
>> This has the obvious advantage, that anyone can use this not only with
>> github but also with his or her own local repository..
>> Then maybe later add more functionality, such as the proposed visual diff
>> or even a simple button to update specific libs from git remote by request
>> of the user..
>>
>> Best regards
>> Simon
>>
>> Am 22. September 2017 03:21:27 MESZ schrieb Oliver Walters <
>> oliver.henry.walt...@gmail.com>:
>>>
>>> Hi all,
>>>
>>> Ok, now that the website integration with the libraries is (pretty much)
>>> done, and the licensing issue seems to be sorted, there is one final puzzle
>>> piece to solve before I'm happy with the state of the libraries for a v5
>>> release.
>>>
>>> *Goal: *Merge all footprint library repositories into a single repo to
>>> solve the ongoing dramas of maintaining 100+ repos.
>>>
>>> *Problem*: The *only* thing standing in the way of just doing this is
>>> that some users like the GitHub plugin and previous instruction is that
>>> this functionality cannot be removed.
>>>
>>> The GitHub plugin functions by downloading a .zip file of a .pretty
>>> repo. If we merge all footprint libs into a single repo with multiple
>>> subdirectories, this will not work anymore (as GitHub dosen't allow you to
>>> download a .zip of a single subdirectory).
>>>
>>> Merging the repos is the *right thing to do*. But how to proceed?
>>>
>>> *Options:*
>>>
>>> *a) Drop github plugin feature, replace with library-download tool*
>>>
>>> I don't think it is a good idea to live-load library data from GitHub (a
>>> lot of other users agree too). It's slow, and a waste of bandwidth to
>>> re-download the libs all the time.
>>>
>>> We drop support for loading libraries direct from GitHub. However, we
>>> add a tool for downloading libraries from GitHub and storing to disk. Users
>>> can update as they like. This can be integrated in KiCad and new users can
>>> run this tool when they first install KICad. This means that no libs need
>>> to distributed with the installer and users can update to latest libs
>>> whenever they want.
>>>
>>> *b) Improve github plugin to allow subdirectory traversal*
>>>
>>> This is difficult and will only result in the plugin being slower. There
>>> are two ways I can see to do this:
>>>
>>> i. Use Git API - tools exist that use this functionality -
>>> https://github.com/KinoLien/gitzip
>>> ii. Use subversion - GitHub actually provides subversion API -
>>> https://www.seanw.org/blog/download-git-repo-subdirectory/
>>>
>>>
>>> *Subversion*
>>>
>>> In either case, I think that using the subversion tool to partially
>>> download the libraries would be a good approach (I assume quicker than
>>> using wget and the GitHub API).
>>>
>>> 1. Does anyone have any experience using the C API for SVN?
>>> 2. The Python bindings are pretty good - https://pypi.python.org/pypi
>>> /svn - and much easier to use. However, can we make the library
>>> download tools dependent on enabling the python plugin?
>>> 3. Is there a way to checkout a subversion remote to memory (to
>>> replicate the functionality of the current GitHub plugin)? If not, I'm not
>>> sure how to approach option b) above.
>>>
>>>
>>> Feedback appreciated. I think that it is very important especially for
>>> new users that this is improved. The GitHub plugin is constantly causing
>>> headaches!
>>>
>>> Cheers,
>>> Oliver
>>>
>>
>> ___
>> Mailing list: https://launchpad.net/~kicad-developers
>> Post to : kicad-developers@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~kicad-developers
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Oliver Walters]

Simon, that would certainly work but svn has the advantage of being able to
pull selective directories from GitHub. You could present the user with a
list of which libraries they actually want to pull down

Disadvantage is that (as you say) non github repos can't be used. Perhaps
that's fine.

On Fri, Sep 22, 2017 at 5:38 PM, Simon Küppers 
wrote:

> Sorry if I don't get it.. Why not just ask the user for a working
> directory and pull the libraries there using actual git?
> This has the obvious advantage, that anyone can use this not only with
> github but also with his or her own local repository..
> Then maybe later add more functionality, such as the proposed visual diff
> or even a simple button to update specific libs from git remote by request
> of the user..
>
> Best regards
> Simon
>
> Am 22. September 2017 03:21:27 MESZ schrieb Oliver Walters <
> oliver.henry.walt...@gmail.com>:
>>
>> Hi all,
>>
>> Ok, now that the website integration with the libraries is (pretty much)
>> done, and the licensing issue seems to be sorted, there is one final puzzle
>> piece to solve before I'm happy with the state of the libraries for a v5
>> release.
>>
>> *Goal: *Merge all footprint library repositories into a single repo to
>> solve the ongoing dramas of maintaining 100+ repos.
>>
>> *Problem*: The *only* thing standing in the way of just doing this is
>> that some users like the GitHub plugin and previous instruction is that
>> this functionality cannot be removed.
>>
>> The GitHub plugin functions by downloading a .zip file of a .pretty repo.
>> If we merge all footprint libs into a single repo with multiple
>> subdirectories, this will not work anymore (as GitHub dosen't allow you to
>> download a .zip of a single subdirectory).
>>
>> Merging the repos is the *right thing to do*. But how to proceed?
>>
>> *Options:*
>>
>> *a) Drop github plugin feature, replace with library-download tool*
>>
>> I don't think it is a good idea to live-load library data from GitHub (a
>> lot of other users agree too). It's slow, and a waste of bandwidth to
>> re-download the libs all the time.
>>
>> We drop support for loading libraries direct from GitHub. However, we add
>> a tool for downloading libraries from GitHub and storing to disk. Users can
>> update as they like. This can be integrated in KiCad and new users can run
>> this tool when they first install KICad. This means that no libs need to
>> distributed with the installer and users can update to latest libs whenever
>> they want.
>>
>> *b) Improve github plugin to allow subdirectory traversal*
>>
>> This is difficult and will only result in the plugin being slower. There
>> are two ways I can see to do this:
>>
>> i. Use Git API - tools exist that use this functionality -
>> https://github.com/KinoLien/gitzip
>> ii. Use subversion - GitHub actually provides subversion API -
>> https://www.seanw.org/blog/download-git-repo-subdirectory/
>>
>>
>> *Subversion*
>>
>> In either case, I think that using the subversion tool to partially
>> download the libraries would be a good approach (I assume quicker than
>> using wget and the GitHub API).
>>
>> 1. Does anyone have any experience using the C API for SVN?
>> 2. The Python bindings are pretty good - https://pypi.python.org/pypi/svn
>> - and much easier to use. However, can we make the library download tools
>> dependent on enabling the python plugin?
>> 3. Is there a way to checkout a subversion remote to memory (to replicate
>> the functionality of the current GitHub plugin)? If not, I'm not sure how
>> to approach option b) above.
>>
>>
>> Feedback appreciated. I think that it is very important especially for
>> new users that this is improved. The GitHub plugin is constantly causing
>> headaches!
>>
>> Cheers,
>> Oliver
>>
>
> ___
> Mailing list: https://launchpad.net/~kicad-developers
> Post to : kicad-developers@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>
>
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Gaurav Juvekar]

Hi,

It would be better to not rely on a GitHub specific SVN API. We can use git 
with --depth=1 for clones and updates (I added this to kicad-library-utils 
download_pretty_libs.py script recently). AFAIK, 3d modules take up the most 
space, so a shallow (depth=1) clone of (all) the symbol and footprint libraries 
could be acceptable.

-- 
Regards,
Gaurav Juvekar

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Simon Küppers]

Sorry if I don't get it.. Why not just ask the user for a working directory and 
pull the libraries there using actual git? 
This has the obvious advantage, that anyone can use this not only with github 
but also with his or her own local repository.. 
Then maybe later add more functionality, such as the proposed visual diff or 
even a simple button to update specific libs from git remote by request of the 
user.. 

Best regards 
Simon 

Am 22. September 2017 03:21:27 MESZ schrieb Oliver Walters 
:
>Hi all,
>
>Ok, now that the website integration with the libraries is (pretty
>much)
>done, and the licensing issue seems to be sorted, there is one final
>puzzle
>piece to solve before I'm happy with the state of the libraries for a
>v5
>release.
>
>*Goal: *Merge all footprint library repositories into a single repo to
>solve the ongoing dramas of maintaining 100+ repos.
>
>*Problem*: The *only* thing standing in the way of just doing this is
>that
>some users like the GitHub plugin and previous instruction is that this
>functionality cannot be removed.
>
>The GitHub plugin functions by downloading a .zip file of a .pretty
>repo.
>If we merge all footprint libs into a single repo with multiple
>subdirectories, this will not work anymore (as GitHub dosen't allow you
>to
>download a .zip of a single subdirectory).
>
>Merging the repos is the *right thing to do*. But how to proceed?
>
>*Options:*
>
>*a) Drop github plugin feature, replace with library-download tool*
>
>I don't think it is a good idea to live-load library data from GitHub
>(a
>lot of other users agree too). It's slow, and a waste of bandwidth to
>re-download the libs all the time.
>
>We drop support for loading libraries direct from GitHub. However, we
>add a
>tool for downloading libraries from GitHub and storing to disk. Users
>can
>update as they like. This can be integrated in KiCad and new users can
>run
>this tool when they first install KICad. This means that no libs need
>to
>distributed with the installer and users can update to latest libs
>whenever
>they want.
>
>*b) Improve github plugin to allow subdirectory traversal*
>
>This is difficult and will only result in the plugin being slower.
>There
>are two ways I can see to do this:
>
>i. Use Git API - tools exist that use this functionality -
>https://github.com/KinoLien/gitzip
>ii. Use subversion - GitHub actually provides subversion API -
>https://www.seanw.org/blog/download-git-repo-subdirectory/
>
>
>*Subversion*
>
>In either case, I think that using the subversion tool to partially
>download the libraries would be a good approach (I assume quicker than
>using wget and the GitHub API).
>
>1. Does anyone have any experience using the C API for SVN?
>2. The Python bindings are pretty good -
>https://pypi.python.org/pypi/svn -
>and much easier to use. However, can we make the library download tools
>dependent on enabling the python plugin?
>3. Is there a way to checkout a subversion remote to memory (to
>replicate
>the functionality of the current GitHub plugin)? If not, I'm not sure
>how
>to approach option b) above.
>
>
>Feedback appreciated. I think that it is very important especially for
>new
>users that this is improved. The GitHub plugin is constantly causing
>headaches!
>
>Cheers,
>Oliver
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Tiger12506]

I'm making more noise again.

Have all options been explored?

Sparse Checkout + clone depth = 1
https://stackoverflow.com/questions/180052/checkout-subdirectories-in-git

Using different branches to differentiate content in git. Seems weird... 
but why not?

https://stackoverflow.com/questions/466303/git-branches-with-completely-different-content

Maybe the svn approach being mentioned is the best, just trying to add 
ideas.


___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Oliver Walters]

Jon,

Using the SVN functionality of GitHub it's possible to do the following:

a) Partial checkout (individual libraries as required)
b) Checkout latest master
c) Checkout tagged releases (e.g. 4.0.7)

It would be "simple" to implement this using Python using the svn bindings.
Would this be a good way to approach this? I'm not sure how to integrate
that into KiCad - is this possible?

If yes, and you can give me some pointers, then I think that (weirdly) the
python / svn approach will allow us to have a pretty powerful library tool
pulling data from GitHub

Thanks,

On Fri, Sep 22, 2017 at 11:26 AM, Jon Evans  wrote:

> I think it would make the most sense to do option (a), possibly also
> leaving open the possibility of changing the updates source location (for
> example, it's conceivable to have a mirror server at kicad-pcb.org that
> contains snapshots of the libraries that are compiled from the Git repo
> whenever it's updated using a Jenkins job).
>
> -Jon
>
> On Thu, Sep 21, 2017 at 9:21 PM, Oliver Walters <
> oliver.henry.walt...@gmail.com> wrote:
>
>> Hi all,
>>
>> Ok, now that the website integration with the libraries is (pretty much)
>> done, and the licensing issue seems to be sorted, there is one final puzzle
>> piece to solve before I'm happy with the state of the libraries for a v5
>> release.
>>
>> *Goal: *Merge all footprint library repositories into a single repo to
>> solve the ongoing dramas of maintaining 100+ repos.
>>
>> *Problem*: The *only* thing standing in the way of just doing this is
>> that some users like the GitHub plugin and previous instruction is that
>> this functionality cannot be removed.
>>
>> The GitHub plugin functions by downloading a .zip file of a .pretty repo.
>> If we merge all footprint libs into a single repo with multiple
>> subdirectories, this will not work anymore (as GitHub dosen't allow you to
>> download a .zip of a single subdirectory).
>>
>> Merging the repos is the *right thing to do*. But how to proceed?
>>
>> *Options:*
>>
>> *a) Drop github plugin feature, replace with library-download tool*
>>
>> I don't think it is a good idea to live-load library data from GitHub (a
>> lot of other users agree too). It's slow, and a waste of bandwidth to
>> re-download the libs all the time.
>>
>> We drop support for loading libraries direct from GitHub. However, we add
>> a tool for downloading libraries from GitHub and storing to disk. Users can
>> update as they like. This can be integrated in KiCad and new users can run
>> this tool when they first install KICad. This means that no libs need to
>> distributed with the installer and users can update to latest libs whenever
>> they want.
>>
>> *b) Improve github plugin to allow subdirectory traversal*
>>
>> This is difficult and will only result in the plugin being slower. There
>> are two ways I can see to do this:
>>
>> i. Use Git API - tools exist that use this functionality -
>> https://github.com/KinoLien/gitzip
>> ii. Use subversion - GitHub actually provides subversion API -
>> https://www.seanw.org/blog/download-git-repo-subdirectory/
>>
>>
>> *Subversion*
>>
>> In either case, I think that using the subversion tool to partially
>> download the libraries would be a good approach (I assume quicker than
>> using wget and the GitHub API).
>>
>> 1. Does anyone have any experience using the C API for SVN?
>> 2. The Python bindings are pretty good - https://pypi.python.org/pypi/svn
>> - and much easier to use. However, can we make the library download tools
>> dependent on enabling the python plugin?
>> 3. Is there a way to checkout a subversion remote to memory (to replicate
>> the functionality of the current GitHub plugin)? If not, I'm not sure how
>> to approach option b) above.
>>
>>
>> Feedback appreciated. I think that it is very important especially for
>> new users that this is improved. The GitHub plugin is constantly causing
>> headaches!
>>
>> Cheers,
>> Oliver
>>
>> ___
>> Mailing list: https://launchpad.net/~kicad-developers
>> Post to : kicad-developers@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~kicad-developers
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Tiger12506]

As a user, I always move to the edge of my seat when this conversation 
comes up. I sit there, waiting for awesome new features like in-program 
diff and pull requesting for kicad libraries, and then I inevitably sit 
back down because (IMHO) kicad wanted to use the github api and start 
wget vs. curl w/openssl license issues or whatever it was that happened 
on the mailing list years ago when the feature got developed, instead of 
trying to use an actual git client to be... well... a git client, and 
because of all the work that went into it, it seems no one wants to 
scrap it.


And I wholeheartedly agree that the libraries should be in one repo, 
i.e. why were they separate to begin with... I've heard that git 
submodules is not awesome, but surely it would apply in this situation? 
You'd get one encompassing repo, and then the submodules that you 
specifically want, no more.


I'm sure everyone thoroughly thought through the design decisions 
involved, it just seems really weird and kicad-style quirky.


Sorry if this email offended anyone, it was only my excitement of 
getting the thought out there, I didn't mean to be rude.

JS

On 9/21/2017 9:26 PM, Jon Evans wrote:
I think it would make the most sense to do option (a), possibly also 
leaving open the possibility of changing the updates source location 
(for example, it's conceivable to have a mirror server at 
kicad-pcb.org  that contains snapshots of the 
libraries that are compiled from the Git repo whenever it's updated 
using a Jenkins job).


-Jon

On Thu, Sep 21, 2017 at 9:21 PM, Oliver Walters 
> wrote:


Hi all,

Ok, now that the website integration with the libraries is (pretty
much) done, and the licensing issue seems to be sorted, there is
one final puzzle piece to solve before I'm happy with the state of
the libraries for a v5 release.

*Goal: *Merge all footprint library repositories into a single
repo to solve the ongoing dramas of maintaining 100+ repos.

*Problem*: The *only* thing standing in the way of just doing this
is that some users like the GitHub plugin and previous instruction
is that this functionality cannot be removed.

The GitHub plugin functions by downloading a .zip file of a
.pretty repo. If we merge all footprint libs into a single repo
with multiple subdirectories, this will not work anymore (as
GitHub dosen't allow you to download a .zip of a single subdirectory).

Merging the repos is the *right thing to do*. But how to proceed?

*Options:*
*
*
/a) Drop github plugin feature, replace with library-download tool/

I don't think it is a good idea to live-load library data from
GitHub (a lot of other users agree too). It's slow, and a waste of
bandwidth to re-download the libs all the time.

We drop support for loading libraries direct from GitHub. However,
we add a tool for downloading libraries from GitHub and storing to
disk. Users can update as they like. This can be integrated in
KiCad and new users can run this tool when they first install
KICad. This means that no libs need to distributed with the
installer and users can update to latest libs whenever they want.

/b) Improve github plugin to allow subdirectory traversal/

This is difficult and will only result in the plugin being slower.
There are two ways I can see to do this:

i. Use Git API - tools exist that use this functionality -
https://github.com/KinoLien/gitzip

ii. Use subversion - GitHub actually provides subversion API -
https://www.seanw.org/blog/download-git-repo-subdirectory/



*Subversion*

In either case, I think that using the subversion tool to
partially download the libraries would be a good approach (I
assume quicker than using wget and the GitHub API).

1. Does anyone have any experience using the C API for SVN?
2. The Python bindings are pretty good -
https://pypi.python.org/pypi/svn
 - and much easier to use.
However, can we make the library download tools dependent on
enabling the python plugin?
3. Is there a way to checkout a subversion remote to memory (to
replicate the functionality of the current GitHub plugin)? If not,
I'm not sure how to approach option b) above.


Feedback appreciated. I think that it is very important especially
for new users that this is improved. The GitHub plugin is
constantly causing headaches!

Cheers,
Oliver

___
Mailing list: https://launchpad.net/~kicad-developers

Post to     : kicad-developers@lists.launchpad.net

Unsubscribe : https://launchpad

Re: [Kicad-developers] GitHub Plugin (my nemesis)

[Jon Evans]

I think it would make the most sense to do option (a), possibly also
leaving open the possibility of changing the updates source location (for
example, it's conceivable to have a mirror server at kicad-pcb.org that
contains snapshots of the libraries that are compiled from the Git repo
whenever it's updated using a Jenkins job).

-Jon

On Thu, Sep 21, 2017 at 9:21 PM, Oliver Walters <
oliver.henry.walt...@gmail.com> wrote:

> Hi all,
>
> Ok, now that the website integration with the libraries is (pretty much)
> done, and the licensing issue seems to be sorted, there is one final puzzle
> piece to solve before I'm happy with the state of the libraries for a v5
> release.
>
> *Goal: *Merge all footprint library repositories into a single repo to
> solve the ongoing dramas of maintaining 100+ repos.
>
> *Problem*: The *only* thing standing in the way of just doing this is
> that some users like the GitHub plugin and previous instruction is that
> this functionality cannot be removed.
>
> The GitHub plugin functions by downloading a .zip file of a .pretty repo.
> If we merge all footprint libs into a single repo with multiple
> subdirectories, this will not work anymore (as GitHub dosen't allow you to
> download a .zip of a single subdirectory).
>
> Merging the repos is the *right thing to do*. But how to proceed?
>
> *Options:*
>
> *a) Drop github plugin feature, replace with library-download tool*
>
> I don't think it is a good idea to live-load library data from GitHub (a
> lot of other users agree too). It's slow, and a waste of bandwidth to
> re-download the libs all the time.
>
> We drop support for loading libraries direct from GitHub. However, we add
> a tool for downloading libraries from GitHub and storing to disk. Users can
> update as they like. This can be integrated in KiCad and new users can run
> this tool when they first install KICad. This means that no libs need to
> distributed with the installer and users can update to latest libs whenever
> they want.
>
> *b) Improve github plugin to allow subdirectory traversal*
>
> This is difficult and will only result in the plugin being slower. There
> are two ways I can see to do this:
>
> i. Use Git API - tools exist that use this functionality -
> https://github.com/KinoLien/gitzip
> ii. Use subversion - GitHub actually provides subversion API -
> https://www.seanw.org/blog/download-git-repo-subdirectory/
>
>
> *Subversion*
>
> In either case, I think that using the subversion tool to partially
> download the libraries would be a good approach (I assume quicker than
> using wget and the GitHub API).
>
> 1. Does anyone have any experience using the C API for SVN?
> 2. The Python bindings are pretty good - https://pypi.python.org/pypi/svn
> - and much easier to use. However, can we make the library download tools
> dependent on enabling the python plugin?
> 3. Is there a way to checkout a subversion remote to memory (to replicate
> the functionality of the current GitHub plugin)? If not, I'm not sure how
> to approach option b) above.
>
>
> Feedback appreciated. I think that it is very important especially for new
> users that this is improved. The GitHub plugin is constantly causing
> headaches!
>
> Cheers,
> Oliver
>
> ___
> Mailing list: https://launchpad.net/~kicad-developers
> Post to : kicad-developers@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp
>
>
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

[Kicad-developers] GitHub Plugin (my nemesis)

[Oliver Walters]

Hi all,

Ok, now that the website integration with the libraries is (pretty much)
done, and the licensing issue seems to be sorted, there is one final puzzle
piece to solve before I'm happy with the state of the libraries for a v5
release.

*Goal: *Merge all footprint library repositories into a single repo to
solve the ongoing dramas of maintaining 100+ repos.

*Problem*: The *only* thing standing in the way of just doing this is that
some users like the GitHub plugin and previous instruction is that this
functionality cannot be removed.

The GitHub plugin functions by downloading a .zip file of a .pretty repo.
If we merge all footprint libs into a single repo with multiple
subdirectories, this will not work anymore (as GitHub dosen't allow you to
download a .zip of a single subdirectory).

Merging the repos is the *right thing to do*. But how to proceed?

*Options:*

*a) Drop github plugin feature, replace with library-download tool*

I don't think it is a good idea to live-load library data from GitHub (a
lot of other users agree too). It's slow, and a waste of bandwidth to
re-download the libs all the time.

We drop support for loading libraries direct from GitHub. However, we add a
tool for downloading libraries from GitHub and storing to disk. Users can
update as they like. This can be integrated in KiCad and new users can run
this tool when they first install KICad. This means that no libs need to
distributed with the installer and users can update to latest libs whenever
they want.

*b) Improve github plugin to allow subdirectory traversal*

This is difficult and will only result in the plugin being slower. There
are two ways I can see to do this:

i. Use Git API - tools exist that use this functionality -
https://github.com/KinoLien/gitzip
ii. Use subversion - GitHub actually provides subversion API -
https://www.seanw.org/blog/download-git-repo-subdirectory/


*Subversion*

In either case, I think that using the subversion tool to partially
download the libraries would be a good approach (I assume quicker than
using wget and the GitHub API).

1. Does anyone have any experience using the C API for SVN?
2. The Python bindings are pretty good - https://pypi.python.org/pypi/svn -
and much easier to use. However, can we make the library download tools
dependent on enabling the python plugin?
3. Is there a way to checkout a subversion remote to memory (to replicate
the functionality of the current GitHub plugin)? If not, I'm not sure how
to approach option b) above.


Feedback appreciated. I think that it is very important especially for new
users that this is improved. The GitHub plugin is constantly causing
headaches!

Cheers,
Oliver
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp