Re: [Kicad-developers] [RFC] Symbol library file format

[Simon Richter]

Hi,

On 03.01.19 19:06, José Ignacio wrote:

> I
> think useful comments to the proposed format should see beyond the
> actual low level representation of the data and talk about the overall
> model being used to store it.

tl;dr: That's a separate discussion.

There are two schools of thought here, one that treats saved data as a
protocol between two black box instances, and one that treats it as a
serialization of the internal state.

Both have advantages and disadvantages. The "protocol" approach allows
changing internals more easily, and gives better compatibility between
versions as changes to the file format have to be made deliberately,
while the "serialization" approach gives us load/save basically for
free, so we need a lot less code.

The "serialization" model also requires us to generate the internal data
structures from a more constrained language like IDL, as the marshaller
needs to know when to follow pointers, and what members of an array are
actually valid.

I'm not sure there are generators that include support for spatial
indexes, though, which is pretty much a requirement for fast rendering,
so this is pretty much impossible at the moment, which places us in
"hand written load/save code" territory anyway.

We have also ignored diff/merge capability so far, which I believe is a
good thing because it cannot really be done on a textual level (schemas
are two-dimensional, PCBs are three-dimensional, so there is no normal
form with a consistent ordering of elements that will make the files
diffable).

I fully expect both the internal model and the file format to change
significantly in the coming years as new features are added. The main
requirement for the file format is that it always needs to be possible
to read older files in some way, and to recognize when a file is newer
than the current parser understands.

It might be a good idea to also have an "extension" mechanism, where we
don't increase the version number when adding a new feature, but rather
mark files that actually use the new feature, so files written by newer
versions that don't use one of the newer functions can be read by older
versions. This would also be another point for "hand written" load/save.

   Simon



signature.asc
Description: OpenPGP digital signature
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] "Fuzzy" antialiasing in eeschema

[Seth Hillbrand]

Am 2019-01-03 22:42, schrieb Alex Lockwood:

Ah, clearly I didn't do a very great job of looking for bug reports.
Thank you.

I wonder whether we could switch to a different antialiasing scheme, or
tweak this one? The antialiased fonts on my system, with lots of 
similar

features to schematics in eeschema, don't do this...


Maybe but that's a very different AA pipeline.  It's not a full-canvas 
AA, so it doesn't need the Sobel step -- it knows where the font edge is 
by definition.  This is extremely precise and doesn't suffer from the 
effect you are seeing.  But this would need to be performed at the draw 
stage prior to compositing, which is not how the architecture is setup 
at the moment.  So definitely not a 5.1 change.


But maybe you have some time during the v6 development? *hint* *hint* 
*nudge* *nudge* :)


___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] "Fuzzy" antialiasing in eeschema

[Alex Lockwood]

Ah, clearly I didn't do a very great job of looking for bug reports.
Thank you.

I wonder whether we could switch to a different antialiasing scheme, or
tweak this one? The antialiased fonts on my system, with lots of similar
features to schematics in eeschema, don't do this...

On Thu, Jan 03, 2019 at 10:10:58PM -0500, Seth Hillbrand wrote:
> Am 2019-01-03 21:14, schrieb Alex Lockwood:
> > I know this is solidly within in realm of minor complaints, but I
> > wonder, is there anything "easy" that can be done about this?
> > 
> > https://misc.c4757p.com/fuzzy-wires.png
> > 
> > A simple answer, I think, is to give eeschema and pcbnew separate
> > settings for antialiasing (as the sort of things pcbnew renders benefit
> > much more from it), but... can anybody explain to someone who doesn't
> > know jack about antialiasing algorithms why this happens? The
> > corners/junctions "spreading out" so far really strikes me as odd from
> > how I understand AA normally working. Are we doing anything...weird?
> 
> Hi Alexis-
> 
> I had a bug report[1] for separating the preferences.  I've been ok with
> pushing this to later because I have found that supersampling 2x works for
> intermediate purposes.  Not as good as fully-separating but better than
> subpixel in eeschema and better than no anti-aliasing in pcbnew.  But that
> might be a combination of my setup, so if you can't find a decent joint
> setting, we should push this back to 5.1
> 
> The "spread" you see is generally the result of the Sobel filter doing
> edge-finding.  It looks for contrast differences and mixes the colors more
> where the contrast is lower.  So, around the junctions, there is more green,
> so there is more weight given to the green when mixing.
> 
> Best-
> Seth
> 
> [1] https://bugs.launchpad.net/kicad/+bug/1808067

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] pcbnew startup from clean

[Alex Lockwood]

Yup. Okay, thanks - I need to pay more attention the bug tracker before
whinging on the list.

On Thu, Jan 03, 2019 at 10:13:25PM -0500, Seth Hillbrand wrote:
> Am 2019-01-03 21:27, schrieb Alex Lockwood:
> > Hey, has anyone started pcbnew with nothing in ~/.config/kicad recently?
> > I just did, and loaded a project, and this is what I got:
> > 
> > https://misc.c4757p.com/wtfpcbnew.png
> > 
> > I had to manually set the canvas from inside pcbnew. Otherwise the
> > rendering was completely screwed up, everything piled on top of
> > everything in a way that I couldn't even really tell what was going on.
> > Uninitialized setting somewhere, maybe? It's entirely possible this is
> > specific to this machine, I also haven't started from clean in a long
> > time. Just curious whether anyone's testing those code paths.
> > 
> > I also had to apply this patch to keep pcbnew from segfaulting when I
> > open menus. Suppose I should probably open a proper bug for this one,
> > but I'm a bit confused by it (it seems like this should have been broken
> > for a long time for everyone, but it...isn't?)
> > 
> > diff --git a/pcbnew/pcb_base_frame.cpp b/pcbnew/pcb_base_frame.cpp
> > index 36ffbd45f..c663f8a60 100644
> > --- a/pcbnew/pcb_base_frame.cpp
> > +++ b/pcbnew/pcb_base_frame.cpp
> > @@ -1189,7 +1189,7 @@ void PCB_BASE_FRAME::OnUpdateSwitchCanvas(
> > wxUpdateUIEvent& aEvent )
> >  for( auto ii: menuList )
> >  {
> >  wxMenuItem* item = menuBar->FindItem( ii.menuId );
> > -if( ii.galType == canvasType )
> > +if( item && ii.galType == canvasType )
> >  item->Check( true );
> >  }
> >  }
> 
> I think that this is the same as [1] that John is working on.  Are you using
> GTK3?
> 
> [1]https://bugs.launchpad.net/kicad/+bug/1809997

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] pcbnew startup from clean

[Seth Hillbrand]

Am 2019-01-03 21:27, schrieb Alex Lockwood:
Hey, has anyone started pcbnew with nothing in ~/.config/kicad 
recently?

I just did, and loaded a project, and this is what I got:

https://misc.c4757p.com/wtfpcbnew.png

I had to manually set the canvas from inside pcbnew. Otherwise the
rendering was completely screwed up, everything piled on top of
everything in a way that I couldn't even really tell what was going on.
Uninitialized setting somewhere, maybe? It's entirely possible this is
specific to this machine, I also haven't started from clean in a long
time. Just curious whether anyone's testing those code paths.

I also had to apply this patch to keep pcbnew from segfaulting when I
open menus. Suppose I should probably open a proper bug for this one,
but I'm a bit confused by it (it seems like this should have been 
broken

for a long time for everyone, but it...isn't?)

diff --git a/pcbnew/pcb_base_frame.cpp b/pcbnew/pcb_base_frame.cpp
index 36ffbd45f..c663f8a60 100644
--- a/pcbnew/pcb_base_frame.cpp
+++ b/pcbnew/pcb_base_frame.cpp
@@ -1189,7 +1189,7 @@ void PCB_BASE_FRAME::OnUpdateSwitchCanvas(
wxUpdateUIEvent& aEvent )
 for( auto ii: menuList )
 {
 wxMenuItem* item = menuBar->FindItem( ii.menuId );
-if( ii.galType == canvasType )
+if( item && ii.galType == canvasType )
 item->Check( true );
 }
 }


I think that this is the same as [1] that John is working on.  Are you 
using GTK3?


[1]https://bugs.launchpad.net/kicad/+bug/1809997

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] "Fuzzy" antialiasing in eeschema

[Seth Hillbrand]

Am 2019-01-03 21:14, schrieb Alex Lockwood:

I know this is solidly within in realm of minor complaints, but I
wonder, is there anything "easy" that can be done about this?

https://misc.c4757p.com/fuzzy-wires.png

A simple answer, I think, is to give eeschema and pcbnew separate
settings for antialiasing (as the sort of things pcbnew renders benefit
much more from it), but... can anybody explain to someone who doesn't
know jack about antialiasing algorithms why this happens? The
corners/junctions "spreading out" so far really strikes me as odd from
how I understand AA normally working. Are we doing anything...weird?


Hi Alexis-

I had a bug report[1] for separating the preferences.  I've been ok with 
pushing this to later because I have found that supersampling 2x works 
for intermediate purposes.  Not as good as fully-separating but better 
than subpixel in eeschema and better than no anti-aliasing in pcbnew.  
But that might be a combination of my setup, so if you can't find a 
decent joint setting, we should push this back to 5.1


The "spread" you see is generally the result of the Sobel filter doing 
edge-finding.  It looks for contrast differences and mixes the colors 
more where the contrast is lower.  So, around the junctions, there is 
more green, so there is more weight given to the green when mixing.


Best-
Seth

[1] https://bugs.launchpad.net/kicad/+bug/1808067

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

[Kicad-developers] pcbnew startup from clean

[Alex Lockwood]

Hey, has anyone started pcbnew with nothing in ~/.config/kicad recently?
I just did, and loaded a project, and this is what I got:

https://misc.c4757p.com/wtfpcbnew.png

I had to manually set the canvas from inside pcbnew. Otherwise the
rendering was completely screwed up, everything piled on top of
everything in a way that I couldn't even really tell what was going on.
Uninitialized setting somewhere, maybe? It's entirely possible this is
specific to this machine, I also haven't started from clean in a long
time. Just curious whether anyone's testing those code paths.

I also had to apply this patch to keep pcbnew from segfaulting when I
open menus. Suppose I should probably open a proper bug for this one,
but I'm a bit confused by it (it seems like this should have been broken
for a long time for everyone, but it...isn't?)

diff --git a/pcbnew/pcb_base_frame.cpp b/pcbnew/pcb_base_frame.cpp
index 36ffbd45f..c663f8a60 100644
--- a/pcbnew/pcb_base_frame.cpp
+++ b/pcbnew/pcb_base_frame.cpp
@@ -1189,7 +1189,7 @@ void PCB_BASE_FRAME::OnUpdateSwitchCanvas( 
wxUpdateUIEvent& aEvent )
 for( auto ii: menuList )
 {
 wxMenuItem* item = menuBar->FindItem( ii.menuId );
-if( ii.galType == canvasType )
+if( item && ii.galType == canvasType )
 item->Check( true );
 }
 }


-- 
Alexis

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

[Kicad-developers] "Fuzzy" antialiasing in eeschema

[Alex Lockwood]

I know this is solidly within in realm of minor complaints, but I
wonder, is there anything "easy" that can be done about this?

https://misc.c4757p.com/fuzzy-wires.png

A simple answer, I think, is to give eeschema and pcbnew separate
settings for antialiasing (as the sort of things pcbnew renders benefit
much more from it), but... can anybody explain to someone who doesn't
know jack about antialiasing algorithms why this happens? The
corners/junctions "spreading out" so far really strikes me as odd from
how I understand AA normally working. Are we doing anything...weird?

-- 
Alexis


___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] grammar.c not found,

[John Beard]

Note, this bug can only be triggered if you do have Lemon installed. Otherwise 
the grammar.c file is just another file.

So make clean is safe unless you have lemon installed! 

Cheers,

John

On 23 December 2018 21:32:50 GMT, John Beard  wrote:
>Hi,
>
>It seems "make clean" might be the trigger here. The target for the
>libeval grammar is done with CMake add_custom_command with an OUTPUT.
>This means the output file will be cleaned on "make clean", but as the
>grammar.c is not then regenerated by default, it will be missing until
>checked out from Git.
>
>Here's a patch to use add_custom_target instead. Any CMake wizards are
>invited to criticise if I have misunderstood. At least from my tests,
>"make clean" no longer removes grammar.c and the target can still be
>used to (re)generate it.
>
>Here's a bug report so we can tie back with the Fixes: line and find
>this issue in future: https://bugs.launchpad.net/kicad/+bug/1809610
>
>Cheers,
>
>John
>On Sun, Dec 23, 2018 at 1:27 AM Nick Østergaard 
>wrote:
>>
>> Hi Lachlan,
>>
>> Are you sure that your workspace is clean? I don't see any issues on
>> any platform.
>>
>> Nick
>>
>> On Sat, 22 Dec 2018 at 06:53, Lachlan Audas 
>wrote:
>> >
>> > Hi,  just pull the lest-est developer build from git-hub for KiCad
>> > and tried compiling it.  But but get this Error, missing file 
>"grammar.c"
>> > >kicad-source-mirror/common/libeval/numeric_evaluator.cpp:36:21:
>fatal error: grammar.c: No such file or directory
>> >  #include "grammar.c"
>> >
>> > the 5.0.2 version has the file,  but not  the developers version
>> > Any one else seen this ?
>> >
>> > Lachlan
>> >
>> > ___
>> > Mailing list: https://launchpad.net/~kicad-developers
>> > Post to : kicad-developers@lists.launchpad.net
>> > Unsubscribe : https://launchpad.net/~kicad-developers
>> > More help   : https://help.launchpad.net/ListHelp
>>
>> ___
>> Mailing list: https://launchpad.net/~kicad-developers
>> Post to : kicad-developers@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~kicad-developers
>> More help   : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

[Kicad-developers] Tool Event Question

[Seth Hillbrand]

Hi Devs-

Quick question for Tom or others familiar with the event handling code.  
In addressing a ratsnest issue[1], I found that the event handler will 
only allow a single action per tool per event.  While we appear to be 
able to queue multiple actions (transitions) for a single tool event, 
the RunMainStack() clears the state transitions immediately after the 
first cofunction is set up.


Is there a technical reason that we do not allow multiple state 
transitions for a single event in the same tool?  If there is, should we 
set up an assert to catch duplicate event transitions when they are 
queued?


-Seth

[1] https://bugs.launchpad.net/kicad/+bug/1809752

___
Mailing list: https://launchpad.net/~kicad-developers
Post to : kicad-developers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kicad-developers
More help   : https://help.launchpad.net/ListHelp

Re: [Kicad-developers] [RFC] Symbol library file format

[Andrew Lutsenko]

> The important thing is what is in the file. If nothing else, S-exp is a
concise way to express this concept during development. Exact format
representation in disk is, right now, bikeshedding.
The important thing is a clean data model and formal IDL like proto helps
with that immensely. If you read through Wayne's google doc comments you
will see how tightly coupled is the discussion of the data model to file
format, which should not be the case. That does not lend itself well to
extensibility and evolution of the data model which is natural part of any
actively developed software.
Thomas pointed out excellent example of issues with current approach in the
other reply. That is very similar to my experience of digging in pcbnew
internals but it's applicable to eeschema as well.

Imagine if pcbnew plugin API was proto based. No more swig quirks for devs
to take care of, no more plugin breakages on every tiny change in Cpp
source, data exchange between main app and plugin is straightforward, they
agree on data model because it has an actual formal definition.

> On the subject of parsers, as opposed to formats, I strongly suggest not
to use the pcbnew one, as it is highly bound up in the data rather than the
syntax. The beauty of sexp is the abstract nature, which lends itself will
to extensible formats. How it is implemented in pcbnew is quite contrary to
that aim and naturally leads to fragility of the parser (for example
unexpected, but syntactically valid, fields can be lethal).

Or you could just use a library that takes care of parsing for you, does it
in all major languages for free, represents formal definition of your data
model and will not break when you add a field.

And to Jose's point
> the S expression parser is already implemented and it works fine, it is
trivial to convert s-expressions to any other data representation you like,
be it, xml, json or whatever comes up next week in NPM
Yeah it works fine but only in cpp and if you want to use it outside of
KiCad codebase you will spend hours reading code to determine actual
format. It is trivial to convert, as long as you put in the aforementioned
hours AND nothing ever changes in the format. Oh and protos have been there
before NPM was a thing and will likely outlive it too (at least one of
thrift or proto definitely will).

> I think useful comments to the proposed format should see beyond the
actual low level representation of the data and talk about the overall
model being used to store it.
I agree completely. Protobufs help with decoupling that.

Regards,
Andrew

On Thu, Jan 3, 2019 at 11:02 AM John Beard  wrote:

> I agree. The important thing is what is in the file. If nothing else,
> S-exp is a concise way to express this concept during development. Exact
> format representation in disk is, right now, bikeshedding.
>
> When we get to that stage, all that is required is that the format is VCS
> friendly and human readable. I suggest that we work out what will be
> represented in the v6 files, then work with a preliminary format at first
> while we implement the kicad data structure handling.
>
> Sexp is probably the simpler path as we have both the pseudo-sexp parser
> used in pcbnew, as well as a "real" one for STEP export (I can't check
> details now, I'm not at my computer).
>
> On the subject of parsers, as opposed to formats, I strongly suggest not
> to use the pcbnew one, as it is highly bound up in the data rather than the
> syntax. The beauty of sexp is the abstract nature, which lends itself will
> to extensible formats. How it is implemented in pcbnew is quite contrary to
> that aim and naturally leads to fragility of the parser (for example
> unexpected, but syntactically valid, fields can be lethal).
>
> Whatever the format, we should strive to separate the handling of the
> syntax of the file and the meaning of the data therein. By doing this, the
> format layer can be swapped and tweaked without reference to the data
> model, and the data model can be changed without touching a byte of the
> format parser/writers.
>
> Cheers,
>
> John
>
>
>
>
> On 3 January 2019 18:06:45 GMT, "José Ignacio" 
> wrote:
>>
>> I think all this babble about data representations to be pointless and
>> counterproductive. the S expression parser is already implemented and it
>> works fine, it is trivial to convert s-expressions to any other data
>> representation you like, be it, xml, json or whatever comes up next week in
>> NPM. The issue with the file format is really to come up with a good data
>> model to represent the objects in kicad, and neither protobufs nor any of
>> the other guys really does anything for us in that area, if anything that
>> is the input that needs to be given to whatever parser generator, or
>> manually generated parser process we choose to utilize. I think useful
>> comments to the proposed format should see beyond the actual low level
>> representation of the data and talk about the overall model being used to
>> 

Re: [Kicad-developers] [RFC] Symbol library file format

[John Beard]

I agree. The important thing is what is in the file. If nothing else, S-exp is 
a concise way to express this concept during development. Exact format 
representation in disk is, right now, bikeshedding.

When we get to that stage, all that is required is that the format is VCS 
friendly and human readable. I suggest that we work out what will be 
represented in the v6 files, then work with a preliminary format at first while 
we implement the kicad data structure handling. 

Sexp is probably the simpler path as we have both the pseudo-sexp parser used 
in pcbnew, as well as a "real" one for STEP export (I can't check details now, 
I'm not at my computer).

On the subject of parsers, as opposed to formats, I strongly suggest not to use 
the pcbnew one, as it is highly bound up in the data rather than the syntax. 
The beauty of sexp is the abstract nature, which lends itself will to 
extensible formats. How it is implemented in pcbnew is quite contrary to that 
aim and naturally leads to fragility of the parser (for example unexpected, but 
syntactically valid, fields can be lethal).

Whatever the format, we should strive to separate the handling of the syntax of 
the file and the meaning of the data therein. By doing this, the format layer 
can be swapped and tweaked without reference to the data model, and the data 
model can be changed without touching a byte of the format parser/writers.

Cheers, 

John




On 3 January 2019 18:06:45 GMT, "José Ignacio"  wrote:
>I think all this babble about data representations to be pointless and
>counterproductive. the S expression parser is already implemented and
>it
>works fine, it is trivial to convert s-expressions to any other data
>representation you like, be it, xml, json or whatever comes up next
>week in
>NPM. The issue with the file format is really to come up with a good
>data
>model to represent the objects in kicad, and neither protobufs nor any
>of
>the other guys really does anything for us in that area, if anything
>that
>is the input that needs to be given to whatever parser generator, or
>manually generated parser process we choose to utilize. I think useful
>comments to the proposed format should see beyond the actual low level
>representation of the data and talk about the overall model being used
>to
>store it.
>
>On Thu, Jan 3, 2019 at 3:37 AM Andrew Lutsenko 
>wrote:
>
>> Hi Martijn,
>> My guess is that most of the complexity is in switching to new data
>model
>> in eeschema, which is going to happen anyway. Since with protobufs
>you
>> don't concern yourself with parsing data the only tricky thing would
>be to
>> seamlessly integrate proto codegen into build process on all
>platforms.
>> Since eeschema data model is going to change without backward
>> compatibility, file compatibility is out of the window too so this is
>good
>> opportunity to change underlying format to something that is better
>in the
>> long run. If we decide to go this route than ideally pcbnew would
>switch to
>> proto at some point too, not necessarily in v6.
>>
>> > Would it be feasible to support both in V6 and let the future tell
>us
>> which one would prevail?
>> What do you mean by prevailing? It's clear at this point that
>> s-expressions are not a winner of global trends while I can guarantee
>that
>> sizable portion of the data in the internet (except probably porn,
>lol) is
>> exchanged in protobufs and thrift since all tech giants use them.
>> If you mean in terms of what KiCad users prefer I don't think vast
>> majority would care either way. Both formats are easy to edit
>directly and
>> protos are much better for devs because codegen takes all the routine
>out
>> of data manipulation.
>> While having both formats coexist is theoretically possible I think
>it
>> will only split the user base since people will pick one and stick to
>it. I
>> think this is the case where devs should be empowered to make the
>choice,
>> same as when switch to s-expressions happened.
>> Of course new version still needs to be able to read old files or at
>least
>> some conversion utility needs to be provided to migrate projects to
>v6 so
>> s-expressions code will not be going away for some time.
>>
>> Andrew
>>
>> On Wed, Jan 2, 2019 at 11:53 PM Martijn Kuipers
>
>> wrote:
>>
>>>
>>>
>>> On 3 Jan 2019, at 04:17, Andrew Lutsenko 
>wrote:
>>>
>>> Wayne,
>>>
>>> > There are some interesting and practical concepts with protobuf
>but it's
>>> functionally a binary storage method which I am opposed to.
>>>
>>> That is a somewhat common misconception because protobufs are
>frequently
>>> used for efficient storage/transfer in binary format. But it's not
>tied to
>>> that format at all, at it's core protobufs are just a way of
>defining well
>>> structured data, nothing more. It comes with bells and whistles like
>>> ability to serialize it in various ways, binary being one of them.
>>>
>>> > Encoding and decoding to a text format would be an acceptable
>solution.
>>> Perfect, here is 

Re: [Kicad-developers] [RFC] Symbol library file format

[Thomas Pointhuber]

Hi,

 

I didn't unterstand the reason for the proposal at the start due to new schematic format, new dependencies,...

 

But thinking about it a bit more it really makes sense to me now. In the past, I wrote importer as well as exporter for the footprint file format, and using protobuf would likely simplify such work quite a bit (no requirement to write sexpr parser/writer, a FORMAL specification of the file format). Have you ever seen the import/export implementation in KiCad itself? I would call it a mess. Locale dependent code which requires workarounds, exporter written with string format which can have issues like missing space between brackets, importers, while written as propper Parser, it should actually be implemented in two stages (stage 1: parse sexpr, stage 2: parse data). Furthermore, you need to have those two completly separated pieces of code to actually produce/understand the same data.

 

In the end, it's the decision of the Developer implementing that stuff. But thinking about the current state, I would say even with sexpr format, the code needs to be refactored quite a bit to be less error-prone and easier to extend. Switching to protobuf should simplify import/export, couples the two parts tightly together, removes the burden to write the actual input/output stuff and simplifies the usage of the file format for 3rd party software (including propper validation). And, in comparsion to the current google-doc, it would be an actual FORMAL specification. I did not even found a normal specification for the .kicad_mod format for example. The best specification is the one where code is generated from. In this case you know there are no implementation differences and it is always up-to-date.

 

Regards,

Thomas

 

Gesendet: Donnerstag, 03. Januar 2019 um 19:06 Uhr
Von: "José Ignacio" 
An: anlutse...@gmail.com
Cc: "KiCad Developers" 
Betreff: Re: [Kicad-developers] [RFC] Symbol library file format


I think all this babble about data representations to be pointless and counterproductive. the S _expression_ parser is already implemented and it works fine, it is trivial to convert s-expressions to any other data representation you like, be it, xml, json or whatever comes up next week in NPM. The issue with the file format is really to come up with a good data model to represent the objects in kicad, and neither protobufs nor any of the other guys really does anything for us in that area, if anything that is the input that needs to be given to whatever parser generator, or manually generated parser process we choose to utilize. I think useful comments to the proposed format should see beyond the actual low level representation of the data and talk about the overall model being used to store it.
 


On Thu, Jan 3, 2019 at 3:37 AM Andrew Lutsenko  wrote:



Hi Martijn,
My guess is that most of the complexity is in switching to new data model in eeschema, which is going to happen anyway. Since with protobufs you don't concern yourself with parsing data the only tricky thing would be to seamlessly integrate proto codegen into build process on all platforms. 

Since eeschema data model is going to change without backward compatibility, file compatibility is out of the window too so this is good opportunity to change underlying format to something that is better in the long run. If we decide to go this route than ideally pcbnew would switch to proto at some point too, not necessarily in v6.

 

> Would it be feasible to support both in V6 and let the future tell us which one would prevail?  

What do you mean by prevailing? It's clear at this point that s-expressions are not a winner of global trends while I can guarantee that sizable portion of the data in the internet (except probably porn, lol) is exchanged in protobufs and thrift since all tech giants use them.

If you mean in terms of what KiCad users prefer I don't think vast majority would care either way. Both formats are easy to edit directly and protos are much better for devs because codegen takes all the routine out of data manipulation.

While having both formats coexist is theoretically possible I think it will only split the user base since people will pick one and stick to it. I think this is the case where devs should be empowered to make the choice, same as when switch to s-expressions happened.

Of course new version still needs to be able to read old files or at least some conversion utility needs to be provided to migrate projects to v6 so s-expressions code will not be going away for some time.

 

Andrew

 


On Wed, Jan 2, 2019 at 11:53 PM Martijn Kuipers  wrote:


 
 

On 3 Jan 2019, at 04:17, Andrew Lutsenko  wrote:
 




Wayne,
 

> There are some interesting and practical concepts with protobuf but it's
functionally a binary storage method which I am opposed to. 

 

That is a somewhat common misconception because protobufs are frequently used for 

Re: [Kicad-developers] [RFC] Symbol library file format

[José Ignacio]

I think all this babble about data representations to be pointless and
counterproductive. the S expression parser is already implemented and it
works fine, it is trivial to convert s-expressions to any other data
representation you like, be it, xml, json or whatever comes up next week in
NPM. The issue with the file format is really to come up with a good data
model to represent the objects in kicad, and neither protobufs nor any of
the other guys really does anything for us in that area, if anything that
is the input that needs to be given to whatever parser generator, or
manually generated parser process we choose to utilize. I think useful
comments to the proposed format should see beyond the actual low level
representation of the data and talk about the overall model being used to
store it.

On Thu, Jan 3, 2019 at 3:37 AM Andrew Lutsenko  wrote:

> Hi Martijn,
> My guess is that most of the complexity is in switching to new data model
> in eeschema, which is going to happen anyway. Since with protobufs you
> don't concern yourself with parsing data the only tricky thing would be to
> seamlessly integrate proto codegen into build process on all platforms.
> Since eeschema data model is going to change without backward
> compatibility, file compatibility is out of the window too so this is good
> opportunity to change underlying format to something that is better in the
> long run. If we decide to go this route than ideally pcbnew would switch to
> proto at some point too, not necessarily in v6.
>
> > Would it be feasible to support both in V6 and let the future tell us
> which one would prevail?
> What do you mean by prevailing? It's clear at this point that
> s-expressions are not a winner of global trends while I can guarantee that
> sizable portion of the data in the internet (except probably porn, lol) is
> exchanged in protobufs and thrift since all tech giants use them.
> If you mean in terms of what KiCad users prefer I don't think vast
> majority would care either way. Both formats are easy to edit directly and
> protos are much better for devs because codegen takes all the routine out
> of data manipulation.
> While having both formats coexist is theoretically possible I think it
> will only split the user base since people will pick one and stick to it. I
> think this is the case where devs should be empowered to make the choice,
> same as when switch to s-expressions happened.
> Of course new version still needs to be able to read old files or at least
> some conversion utility needs to be provided to migrate projects to v6 so
> s-expressions code will not be going away for some time.
>
> Andrew
>
> On Wed, Jan 2, 2019 at 11:53 PM Martijn Kuipers 
> wrote:
>
>>
>>
>> On 3 Jan 2019, at 04:17, Andrew Lutsenko  wrote:
>>
>> Wayne,
>>
>> > There are some interesting and practical concepts with protobuf but it's
>> functionally a binary storage method which I am opposed to.
>>
>> That is a somewhat common misconception because protobufs are frequently
>> used for efficient storage/transfer in binary format. But it's not tied to
>> that format at all, at it's core protobufs are just a way of defining well
>> structured data, nothing more. It comes with bells and whistles like
>> ability to serialize it in various ways, binary being one of them.
>>
>> > Encoding and decoding to a text format would be an acceptable solution.
>> Perfect, here is example of built in proto text encoder, it resembles
>> JSON in that it uses curly braces to encase submessages but doesn't abuse
>> punctuation marks unnecessarily.
>>
>> user_collection {
>>   description = "my default users"
>>   users {
>> key: "user_1234"
>> value {
>>   handle: "winniepoo"
>>   paid_membership: true
>> }
>>   }
>>   users {
>> key: "user_9b27"
>> value {
>>   handle: "smokeybear"
>> }
>>   }}
>>
>> > There is also the issue of learning curve and another build dependency.
>> Yes, that is inevitable but benefits I outlined in my earlier email are
>> too significant to overlook in my opinion.
>>
>> > S-expr is not at all like XML at least not in terms of readability.
>> It actually is a lot like XML, just less pointy brackets. Same arbitrary
>> distinctions between attributes (which are not even named in S-expr, which
>> adds to confusion when glancing at the file) and subfields.
>>
>> Here is real example:
>> (fp_text value 330K (at 0 -1.75) (layer B.Fab)
>>   (effects (font (size 1 1) (thickness 0.15)) (justify mirror))
>> )
>> or
>> (pad 2 smd rect (at 0.95 0 180) (size 0.7 1.3) (layers B.Cu B.Paste
>> B.Mask)
>>   (net 95 "Net-(R17-Pad2)"))
>>
>> Without reading file format docs and/or source code I have no idea why
>> some data is in subfields and some data is in some special fixed order on
>> the same level as it's container.
>> It's also very easy to confuse "330K" being related to "value" when in
>> fact "value" is field name and "330K" is the value. In proto that would
>> 

Re: [Kicad-developers] [RFC] Symbol library file format

[Andrew Lutsenko]

Hi Martijn,
My guess is that most of the complexity is in switching to new data model
in eeschema, which is going to happen anyway. Since with protobufs you
don't concern yourself with parsing data the only tricky thing would be to
seamlessly integrate proto codegen into build process on all platforms.
Since eeschema data model is going to change without backward
compatibility, file compatibility is out of the window too so this is good
opportunity to change underlying format to something that is better in the
long run. If we decide to go this route than ideally pcbnew would switch to
proto at some point too, not necessarily in v6.

> Would it be feasible to support both in V6 and let the future tell us
which one would prevail?
What do you mean by prevailing? It's clear at this point that s-expressions
are not a winner of global trends while I can guarantee that sizable
portion of the data in the internet (except probably porn, lol) is
exchanged in protobufs and thrift since all tech giants use them.
If you mean in terms of what KiCad users prefer I don't think vast majority
would care either way. Both formats are easy to edit directly and protos
are much better for devs because codegen takes all the routine out of data
manipulation.
While having both formats coexist is theoretically possible I think it will
only split the user base since people will pick one and stick to it. I
think this is the case where devs should be empowered to make the choice,
same as when switch to s-expressions happened.
Of course new version still needs to be able to read old files or at least
some conversion utility needs to be provided to migrate projects to v6 so
s-expressions code will not be going away for some time.

Andrew

On Wed, Jan 2, 2019 at 11:53 PM Martijn Kuipers 
wrote:

>
>
> On 3 Jan 2019, at 04:17, Andrew Lutsenko  wrote:
>
> Wayne,
>
> > There are some interesting and practical concepts with protobuf but it's
> functionally a binary storage method which I am opposed to.
>
> That is a somewhat common misconception because protobufs are frequently
> used for efficient storage/transfer in binary format. But it's not tied to
> that format at all, at it's core protobufs are just a way of defining well
> structured data, nothing more. It comes with bells and whistles like
> ability to serialize it in various ways, binary being one of them.
>
> > Encoding and decoding to a text format would be an acceptable solution.
> Perfect, here is example of built in proto text encoder, it resembles JSON
> in that it uses curly braces to encase submessages but doesn't abuse
> punctuation marks unnecessarily.
>
> user_collection {
>   description = "my default users"
>   users {
> key: "user_1234"
> value {
>   handle: "winniepoo"
>   paid_membership: true
> }
>   }
>   users {
> key: "user_9b27"
> value {
>   handle: "smokeybear"
> }
>   }}
>
> > There is also the issue of learning curve and another build dependency.
> Yes, that is inevitable but benefits I outlined in my earlier email are
> too significant to overlook in my opinion.
>
> > S-expr is not at all like XML at least not in terms of readability.
> It actually is a lot like XML, just less pointy brackets. Same arbitrary
> distinctions between attributes (which are not even named in S-expr, which
> adds to confusion when glancing at the file) and subfields.
>
> Here is real example:
> (fp_text value 330K (at 0 -1.75) (layer B.Fab)
>   (effects (font (size 1 1) (thickness 0.15)) (justify mirror))
> )
> or
> (pad 2 smd rect (at 0.95 0 180) (size 0.7 1.3) (layers B.Cu B.Paste
> B.Mask)
>   (net 95 "Net-(R17-Pad2)"))
>
> Without reading file format docs and/or source code I have no idea why
> some data is in subfields and some data is in some special fixed order on
> the same level as it's container.
> It's also very easy to confuse "330K" being related to "value" when in
> fact "value" is field name and "330K" is the value. In proto that would
> look like this:
>
> fp_text {
>   name: "value"
>   value: "330K"
>   at {
> x: 0
> y: -1.75
>   }
>   layer: "B.Fab"
>   effects {
> ...
>   }
> }
>
> But I do agree on the point that all markup formats have their downsides.
> For example json/yaml or proto text format would be less dense per line,
> however that may be an upside when looked at it from version control and
> diff-ing perspective.
>
> > To me self documenting means that the file format doesn't even require a
> > document to explain it's contents.  It should be self evident from the
> > contents of the file.
> What I meant by self documenting was that the document describing the
> format and document implementing the format (actual source code) was the
> same.
> But even as evident from example above, proto text format yields file
> contents that are pretty well self documented too.
>
> > Changing file formats would be a
> > benefit but why would we need to do that?  If we have a human readable
> >