Just to keep this discussion alive. There is a prototype now which allows KiCad to talk to an external REST API.
https://gitlab.com/kicad/code/kicad/-/merge_requests/1679 On Friday, 14 July 2023 at 2:04:46 am UTC+10 [email protected] wrote: > One comment on this: > > *most of these data providers would be unhappy with KiCad providing an > easy way for users to scrape their data, as it could cost them a lot of > money* > > Actually, Mouser , Digikey, Element14, among a few other already provide > this since a long time for free but with a limit on maximum number of > calls or records retrieved per day, based on the user's registered > account. Though this quota is sufficient say ~100 searches per day. > Recently LCSC stopped their free service and only allows customers with a > VERY VERY LARGE order amount history. (think CM's), Other retailers > require paid subscriptions. These services often involve OAuth2. and > that probably is beyond scope for KicAD and even many users as it would > need to manage call backs, client secrets, access token, refresh tokens > etc...That is a complex affair. > > Therefor I documented in the proposals that an "intermediate" service > running locally or on a remote server implemented by the user (not kicad) > to handle the transformation > between the call to the API and the service's data and also in there > manage authentication (call backs, client credentials, access token and > refresh tokens etc..). This part could e provided by external open source > projects and out of scope for KiCad. KiCad just makes the API call to that > "intermediate" service based on the KicaD API spec. > > > > On Thursday, July 13, 2023 at 11:45:39 PM UTC+8 Symdeb wrote: > >> The word "instead" was probably a bit poorly chosen. Surely, retaining >> ODBC makes sense to make since it's already there and people are using that. >> No objection on plugins or any variant of solution. As any proposal , the >> document was just for feedback, discussion, consideration. Nothing set in >> stone. >> It was kept as generic as possible w/o restricting the type of >> implementation. >> >> >> >> >> >> >> >> On Thursday, July 13, 2023 at 9:55:02 PM UTC+8 [email protected] wrote: >> >>> There are a lot of different concepts and ideas here being thrown around. >>> >>> First of all, to reiterate what I said on the linked issue: No REST API >>> will be accepted "instead of" ODBC. The existing ODBC system will continue >>> to exist as-is to support users for whom it is a better solution than REST. >>> >>> Second, the scope of any project being developed by someone who hasn't >>> worked on KiCad before really should be limited, as it will take time for >>> someone to learn the codebase and build a good working relationship with >>> the rest of the team. To that end, I do not think it is in scope to >>> consider any functionality not already supported by the KiCad library >>> system. So, change management / version control are _not_ something I >>> would worry about at this point when defining the API interface with KiCad. >>> >>> If someone wants to build a change management / version control system >>> outside of KiCad (as part of whatever web service is exposing this proposed >>> REST API) that is fine. >>> >>> If you or anyone else is serious about pursuing this, I would recommend >>> the next step is to continue to build out the API proposal document, based >>> on APIs required to build a KiCad symbol library plugin. Once we are in >>> agreement on that, a proof-of-concept could be built for testing and review. >>> >>> -Jon >>> >>> On Thu, Jul 13, 2023 at 12:46 AM Symdeb <[email protected]> wrote: >>> >>>> There were several conversations in the past on Kicad Info, Inventree >>>> Github and Kicad gitlab. >>>> Here is a functional specification proposal for an API instead of ODBC. >>>> This built on top the proposed search for parts feature on gitlab , as >>>> follows >>>> >>>> 1. Searching for parts from any resource . The API implementation >>>> needs to be provided, not in scope. Specification adds request/response >>>> proposal >>>> 2. Change (version) management between CAD and remote systems for >>>> footprints, symbols, models. Specification adds request/response >>>> proposals >>>> 3. Part to symbol/footprint/model mapping between remote system and >>>> the CAD. Specification adds request/response proposals ad version >>>> control >>>> logic. >>>> >>>> "remote system" term is used here because it could be a PDM, PLM , home >>>> brewed python server script to a local or remote DB etc.. >>>> References: >>>> https://github.com/inventree/InvenTree/discussions/4133 >>>> >>>> https://docs.google.com/document/d/1qXKEa9Djz1VnEzH7jK0rJ1p9Vu82tq4p_m2zBuNMG6k/edit#heading=h.fyuo1bpfgubj >>>> https://gitlab.com/kicad/code/kicad/-/issues/12027 >>>> >>>> >>>> >>>> On Tuesday, July 26, 2022 at 1:52:14 AM UTC+8 [email protected] wrote: >>>> >>>>> Yes, the database library would be a type of library format (exact UX >>>>> is not yet determined). >>>>> >>>>> -Jon >>>>> >>>>> On Thu, Jul 21, 2022 at 7:00 PM Andre Iwers <[email protected]> wrote: >>>>> >>>>>> I assume that this will be some sort of library format and users >>>>>> would be able to select this in here right? >>>>>> [image: Screenshot 2022-07-22 084207.png] >>>>>> >>>>>> I would be more than happy to use a non-compiled cross-platform >>>>>> plugin system. What's important to me, and what makes my and my >>>>>> colleagues >>>>>> life easier, is that it is well integrated and offers a unified >>>>>> interface; >>>>>> similar to what I showed here: >>>>>> https://gitlab.com/kicad/code/kicad/-/issues/12027 >>>>>> >>>>>> Apart from this, I am more than happy to use kicad's internal tools >>>>>> to create symbols and footprints and place those using the currently >>>>>> available facilities. In fact, currently our meta data also contains the >>>>>> right footprint string which reduced accidental placement of the wrong >>>>>> footprints a fair bit; it's also a QM thing. >>>>>> >>>>>> On Thursday, 21 July 2022 at 11:11:48 pm UTC+10 [email protected] >>>>>> wrote: >>>>>> >>>>>>> The ODBC interface is designed to support pulling component >>>>>>> libraries from the database, not adding metadata to parts that are from >>>>>>> other libraries. So it's a slightly different (but related) use case. >>>>>>> At >>>>>>> the moment there are no plans to support anything other than ODBC. >>>>>>> >>>>>>> The use case of adding metadata to existing parts that come from >>>>>>> other libraries is what you seem to be heading towards. This one I >>>>>>> would >>>>>>> recommend we look at solving with a cross-platform plugin system of >>>>>>> some >>>>>>> sort. >>>>>>> >>>>>>> -Jon >>>>>>> >>>>>>> On Thu, Jul 21, 2022 at 1:05 AM Andre Iwers <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Jon & Seth, >>>>>>>> >>>>>>>> Thanks for sharing your thoughts! >>>>>>>> >>>>>>>> Some thoughts here from my side: >>>>>>>> >>>>>>>> 1. Storing part metadata in a private database -- I have >>>>>>>> started working on this feature (#7436 >>>>>>>> <https://gitlab.com/kicad/code/kicad/-/issues/7436>). The >>>>>>>> scope of this one is that the database is SQL-like and >>>>>>>> ODBC-compatible. >>>>>>>> - I am not sure if this relying on ODBC is flexible enough. >>>>>>>> In my case it would not be. I mostly work with either Navision >>>>>>>> or Inventree >>>>>>>> which stock data is managed by other people. In my case I rely >>>>>>>> heavily on >>>>>>>> their stock/metadata and I don't have direct access to the >>>>>>>> source. It's >>>>>>>> read-only. Having an ODBC compatible interface would mean that >>>>>>>> every >>>>>>>> software you'd need to work with must support that, or am I >>>>>>>> wrong here? >>>>>>>> - My approach tried to avoid exactly that. I have a driver >>>>>>>> which does all the translation work between kicad and the >>>>>>>> inventory system >>>>>>>> and the user can design that plugin as needed. Kicad would only >>>>>>>> impIement >>>>>>>> that once and that's it; there is no need to touch the source >>>>>>>> again when >>>>>>>> changing the plugins itself. >>>>>>>> - I admit that a compiled library might not be ideal as >>>>>>>> kicad is cross-platform, however, one could use a simple >>>>>>>> inventory_plugin.py file or archive which is digested by kicad >>>>>>>> and then use >>>>>>>> a python interpreter. >>>>>>>> >>>>>>>> >>>>>>>> 1. Retrieving live part metadata from a public data source >>>>>>>> (Mouser/Digi-Key/Octopart/JLC/etc) and applying it to >>>>>>>> already-placed >>>>>>>> symbols (or newly-placed). >>>>>>>> - The above approach could be used here too. Digikey >>>>>>>> provides API access and everyone can get their own API key. >>>>>>>> >>>>>>>> >>>>>>>> Please let me know what you think. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Andre >>>>>>>> >>>>>>>> On Tuesday, 19 July 2022 at 11:47:45 pm UTC+10 [email protected] >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Just to add on to what Jon has already said: >>>>>>>>> >>>>>>>>> KiCad is a cross-platform application, so any solution needs to >>>>>>>>> similarly be cross-platform. If the target for the external plugin >>>>>>>>> is to >>>>>>>>> allow users and companies to develop their own interfaces that can be >>>>>>>>> used >>>>>>>>> in KiCad, then this needs to be done in a scripting language. >>>>>>>>> >>>>>>>>> There are a couple reasons for this. The big one is that MacOS >>>>>>>>> cannot load a binary plugin from outside the signed application >>>>>>>>> without >>>>>>>>> triggering GateKeeper and all of the badness that comes with that. >>>>>>>>> >>>>>>>>> We can build/distribute binary plugins (like the 3d plugins) that >>>>>>>>> are included in our codebase because these get signed with the rest >>>>>>>>> of the >>>>>>>>> application. So, if the target is to allow ODBC interaction, we can >>>>>>>>> probably support a binary plugin interface that explicitly does that. >>>>>>>>> I >>>>>>>>> think that this is (#7436 >>>>>>>>> <https://gitlab.com/kicad/code/kicad/-/issues/7436>) >>>>>>>>> >>>>>>>>> Seth >>>>>>>>> >>>>>>>>> On Tue, Jul 19, 2022 at 6:05 AM Jon Evans <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Andre, >>>>>>>>>> >>>>>>>>>> I think it would be best to have two different discussions here, >>>>>>>>>> first about the idea and concept, and second about your proposed >>>>>>>>>> implementation. >>>>>>>>>> I wanted to lead with this because I think there is a tendency in >>>>>>>>>> discussions to mix the two together unnecessarily. >>>>>>>>>> >>>>>>>>>> This is a real need and something that we've already started >>>>>>>>>> working on related things for, although in a slightly different >>>>>>>>>> direction >>>>>>>>>> than you describe. There are two different but related concepts >>>>>>>>>> here: >>>>>>>>>> >>>>>>>>>> 1) Storing part metadata in a private database -- I have started >>>>>>>>>> working on this feature (#7436 >>>>>>>>>> <https://gitlab.com/kicad/code/kicad/-/issues/7436>). The scope >>>>>>>>>> of this one is that the database is SQL-like and ODBC-compatible. >>>>>>>>>> >>>>>>>>>> 2) Retrieving live part metadata from a public data source >>>>>>>>>> (Mouser/Digi-Key/Octopart/JLC/etc) and applying it to already-placed >>>>>>>>>> symbols (or newly-placed). >>>>>>>>>> >>>>>>>>>> (1) can obviously be built into KiCad and so it will be. (2) is >>>>>>>>>> more challenging -- most of these data providers would be unhappy >>>>>>>>>> with >>>>>>>>>> KiCad providing an easy way for users to scrape their data, as it >>>>>>>>>> could >>>>>>>>>> cost them a lot of money. There have been discussions with various >>>>>>>>>> data >>>>>>>>>> vendors about an integration approach, but in general this is >>>>>>>>>> something >>>>>>>>>> that needs to be negotiated per-vendor. >>>>>>>>>> >>>>>>>>>> Because of this, I agree that the right approach is to build a >>>>>>>>>> framework rather than an implementation, which leaves e.g. Digi-Key >>>>>>>>>> open to >>>>>>>>>> providing their own KiCad data plugin if they wish to make their >>>>>>>>>> data >>>>>>>>>> available to KiCad users in this way. >>>>>>>>>> >>>>>>>>>> This leads me to the implementation -- I'm not sure that creating >>>>>>>>>> a new binary plugin interface is the approach we should take for >>>>>>>>>> this. We >>>>>>>>>> have been investing in the Python plugin ecosystem elsewhere, and >>>>>>>>>> this >>>>>>>>>> seems like an ideal fit for Python: people creating datasource >>>>>>>>>> plugins can >>>>>>>>>> distribute a single plugin that runs on any platform KiCad is >>>>>>>>>> supported in, >>>>>>>>>> and there are fewer stability and security concerns with loading an >>>>>>>>>> arbitrary Python plugin than with an arbitrary compiled binary. >>>>>>>>>> >>>>>>>>>> For the workflow you describe: this makes sense in the use-case >>>>>>>>>> of adding metadata from an external source to parts that were >>>>>>>>>> already >>>>>>>>>> placed. The Database Libraries workflow will be oriented more about >>>>>>>>>> placing symbols with all their metadata from the start. Perhaps it >>>>>>>>>> would >>>>>>>>>> be possible to have this proposed metadata plugins interface work >>>>>>>>>> either at >>>>>>>>>> initial placement or afterwards? >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> -Jon >>>>>>>>>> >>>>>>>>>> On Mon, Jul 18, 2022 at 6:53 PM Andre Iwers <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi There! >>>>>>>>>>> >>>>>>>>>>> I constantly run into issues where I need to add all sorts of >>>>>>>>>>> metadata to my sometimes 100+ parts schematics which are saved in >>>>>>>>>>> an >>>>>>>>>>> external inventory system. Initially doing it by hand I grew tired >>>>>>>>>>> of it >>>>>>>>>>> and decided to whip up a slightly less laborious way using a plugin >>>>>>>>>>> approach which is quite flexible in terms of datasources. Also >>>>>>>>>>> updating >>>>>>>>>>> those is even worse. >>>>>>>>>>> >>>>>>>>>>> I was interested if this is something Kicad would be interested >>>>>>>>>>> in taking over for the broader public. I hooked it into the 'Symbol >>>>>>>>>>> Properties' or 'Symbol Fields Table' dialog for ease of use and >>>>>>>>>>> touched >>>>>>>>>>> that existing code only marginally. >>>>>>>>>>> >>>>>>>>>>> <https://mail.google.com/mail/u/1/#m_-4237403401938012105_expected-behavior>Quick >>>>>>>>>>> >>>>>>>>>>> Overview: >>>>>>>>>>> >>>>>>>>>>> When I have selected a schematic symbol from the standard >>>>>>>>>>> library, I'd like to be able to open the 'Symbol Properties' or >>>>>>>>>>> 'Symbol >>>>>>>>>>> Fields Table' to fetch part information from an external source. >>>>>>>>>>> >>>>>>>>>>> Idea: Eeschema would simply provide a framework which connects >>>>>>>>>>> to one or more plugins and the plugins themselves would be provided >>>>>>>>>>> by the >>>>>>>>>>> community as .dll or .so which then are saved somewhere on the >>>>>>>>>>> computer and >>>>>>>>>>> simply loaded when needed. The plugin can be programmed in any >>>>>>>>>>> language >>>>>>>>>>> such as python, java, c# etc. and would only need to honour the >>>>>>>>>>> defined C >>>>>>>>>>> wrapper interface. Plugins which don't adhere to that are simply >>>>>>>>>>> ignored. >>>>>>>>>>> >>>>>>>>>>> I have created a first draft of that framework to solve a >>>>>>>>>>> laborious task I face once in a while and am wondering if this is >>>>>>>>>>> something >>>>>>>>>>> that is generally useful for the broader public. I also programmed >>>>>>>>>>> a plugin >>>>>>>>>>> for Inventree which is used here as part database. >>>>>>>>>>> >>>>>>>>>>> In general: the framework is intended to use multiple plugins at >>>>>>>>>>> the same time so that a user could potentially query for example >>>>>>>>>>> Mouser, >>>>>>>>>>> Digikey, Navision and Inventree at the same time. >>>>>>>>>>> >>>>>>>>>>> <https://mail.google.com/mail/u/1/#m_-4237403401938012105_the-workflow>The >>>>>>>>>>> >>>>>>>>>>> workflow: >>>>>>>>>>> <https://mail.google.com/mail/u/1/#m_-4237403401938012105_a-brand-new-project-would-be-to>A >>>>>>>>>>> >>>>>>>>>>> brand new project would be to: >>>>>>>>>>> >>>>>>>>>>> - select a symbol from the standard kicad library >>>>>>>>>>> - open plugin search and search for the specific part >>>>>>>>>>> - transfer all metadata automatically into your schematic >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> <https://mail.google.com/mail/u/1/#m_-4237403401938012105_existing-project>Existing >>>>>>>>>>> >>>>>>>>>>> project: >>>>>>>>>>> >>>>>>>>>>> - Open fields editor and bulk apply metadata using the above >>>>>>>>>>> mentioned approach. >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "KiCad Developers" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>> https://groups.google.com/a/kicad.org/d/msgid/devlist/CABQVKmKZAQtXZ-gdQOJiHGH_vJmwfU5Ho%3DCAaMtM_YD2PbZteQ%40mail.gmail.com >>>>>>>>>>> >>>>>>>>>>> <https://groups.google.com/a/kicad.org/d/msgid/devlist/CABQVKmKZAQtXZ-gdQOJiHGH_vJmwfU5Ho%3DCAaMtM_YD2PbZteQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>>> . >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "KiCad Developers" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected]. >>>>>>>>>> >>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/a/kicad.org/d/msgid/devlist/CA%2BqGbCBOVz_8CJLG5spNgg%3D%3D6RF4s390abE_LW8wQeeE5-A%3DdQ%40mail.gmail.com >>>>>>>>>> >>>>>>>>>> <https://groups.google.com/a/kicad.org/d/msgid/devlist/CA%2BqGbCBOVz_8CJLG5spNgg%3D%3D6RF4s390abE_LW8wQeeE5-A%3DdQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> [image: KiCad Services Corporation Logo] >>>>>>>>> Seth Hillbrand >>>>>>>>> *Lead Developer* >>>>>>>>> +1-530-302-5483 <(530)%20302-5483> >>>>>>>>> Long Beach, CA >>>>>>>>> www.kipro-pcb.com [email protected] >>>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "KiCad Developers" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> >>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/a/kicad.org/d/msgid/devlist/2838e643-a03c-4cdb-8853-3228bfbd19d0n%40kicad.org >>>>>>>> >>>>>>>> <https://groups.google.com/a/kicad.org/d/msgid/devlist/2838e643-a03c-4cdb-8853-3228bfbd19d0n%40kicad.org?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- You received this message because you are subscribed to the Google Groups "KiCad Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/kicad.org/d/msgid/devlist/d390661d-a4d3-4c05-a063-1a4256833de2n%40kicad.org.
