Re: Reviving the hardware census

[Nathaniel McCallum]

It isn't documented in F27, but it does work. However, we probably
want at least this patch:
https://github.com/lyonel/lshw/commit/135a853c60582b14c5b67e5cd988a8062d9896f4

On Wed, Nov 8, 2017 at 4:28 PM, Don Zickus  wrote:
> On Wed, Nov 08, 2017 at 04:09:26PM -0500, Nathaniel McCallum wrote:
>> I just looked at the code for lshw. The master branch already supports
>> JSON. We just need them to release it.
>
> Eh?  'lshw -json' doesn't work for you?  I thought that was a supported
> output for a while now.  At least it works on my F27 box, but I think we
> have it running successfully under RHEL-7 too.
>
> Cheers,
> Don
>
>>
>> On Wed, Nov 8, 2017 at 3:23 PM, Don Zickus  wrote:
>> > On Wed, Nov 08, 2017 at 03:16:24PM -0500, Nathaniel McCallum wrote:
>> >> I just played around with lshw a bit. We should totally make it export
>> >> JSON. We can then submit this directly (as one census plugin).
>> >
>> > Yes, that is how we use it to update hardware info internally to our Beaker
>> > instance. :-)
>> >
>> > Cheers,
>> > Don
>> >
>> >>
>> >> On Wed, Nov 8, 2017 at 12:34 PM, Don Zickus  wrote:
>> >> > On Tue, Nov 07, 2017 at 10:49:02PM +, Jeremy Cline wrote:
>> >> >> Hey folks,
>> >> >>
>> >> >> For some time now, Fedora has operated without a database of hardware
>> >> >> users have. Smolt, the old hardware database, was retired in 2012[0] 
>> >> >> and
>> >> >> its intended successor[1] was never deployed by Fedora Infrastructure.
>> >> >>
>> >> >> It would be nice to have a hardware database, so I (and hopefully some
>> >> >> others) would like to get Census up and running for Fedora. Before we
>> >> >> look at deploying Census, however, it would be good to make sure it has
>> >> >> everything we need.
>> >> >>
>> >> >> Census has client plugins to collect information[2]. At the moment, it
>> >> >> has plugins for:
>> >> >>
>> >> >> * The vendor, device, subsystem_vendor, subsystem_device, and class 
>> >> >> from
>> >> >>   each PCI device
>> >> >>
>> >> >> * The idVendor, idProduct, bcdDevice, and bDeviceClass for USB devices
>> >> >>   as well as the bInterfaceClass, bInterfaceSubClass, and
>> >> >>   bInterfaceProtocol for each interface
>> >> >>
>> >> >> * The contents of /etc/os-release
>> >> >>
>> >> >> * All the RPMs installed on a system
>> >> >>
>> >> >> Other than the drivers bound to the PCI and USB devices (which is an
>> >> >> open PR[3]), what else would be good to collect?
>> >> >>
>> >> >> [0] https://fedoraproject.org/wiki/Smolt_retirement
>> >> >> [1] https://github.com/npmccallum/census
>> >> >> [2] https://github.com/npmccallum/census/blob/master/client/plugins/
>> >> >> [3] https://github.com/npmccallum/census/pull/3
>> >> >
>> >> > Internally, we have been focusing on using 'lshw' as the tool that 
>> >> > provides
>> >> > all that info and handles all the arch funkiness (and includes 
>> >> > firmware).
>> >> > If there is anything missing, we have tried to push upstream to that
>> >> > project.
>> >> >
>> >> > Would that cover a lot of the info you are looking for?
>> >> >
>> >> > Cheers,
>> >> > Don
>> >> ___
>> >> kernel mailing list -- kernel@lists.fedoraproject.org
>> >> To unsubscribe send an email to kernel-le...@lists.fedoraproject.org
>> ___
>> kernel mailing list -- kernel@lists.fedoraproject.org
>> To unsubscribe send an email to kernel-le...@lists.fedoraproject.org
___
kernel mailing list -- kernel@lists.fedoraproject.org
To unsubscribe send an email to kernel-le...@lists.fedoraproject.org

Re: Reviving the hardware census

[Don Zickus]

On Wed, Nov 08, 2017 at 04:09:26PM -0500, Nathaniel McCallum wrote:
> I just looked at the code for lshw. The master branch already supports
> JSON. We just need them to release it.

Eh?  'lshw -json' doesn't work for you?  I thought that was a supported
output for a while now.  At least it works on my F27 box, but I think we
have it running successfully under RHEL-7 too.

Cheers,
Don

> 
> On Wed, Nov 8, 2017 at 3:23 PM, Don Zickus  wrote:
> > On Wed, Nov 08, 2017 at 03:16:24PM -0500, Nathaniel McCallum wrote:
> >> I just played around with lshw a bit. We should totally make it export
> >> JSON. We can then submit this directly (as one census plugin).
> >
> > Yes, that is how we use it to update hardware info internally to our Beaker
> > instance. :-)
> >
> > Cheers,
> > Don
> >
> >>
> >> On Wed, Nov 8, 2017 at 12:34 PM, Don Zickus  wrote:
> >> > On Tue, Nov 07, 2017 at 10:49:02PM +, Jeremy Cline wrote:
> >> >> Hey folks,
> >> >>
> >> >> For some time now, Fedora has operated without a database of hardware
> >> >> users have. Smolt, the old hardware database, was retired in 2012[0] and
> >> >> its intended successor[1] was never deployed by Fedora Infrastructure.
> >> >>
> >> >> It would be nice to have a hardware database, so I (and hopefully some
> >> >> others) would like to get Census up and running for Fedora. Before we
> >> >> look at deploying Census, however, it would be good to make sure it has
> >> >> everything we need.
> >> >>
> >> >> Census has client plugins to collect information[2]. At the moment, it
> >> >> has plugins for:
> >> >>
> >> >> * The vendor, device, subsystem_vendor, subsystem_device, and class from
> >> >>   each PCI device
> >> >>
> >> >> * The idVendor, idProduct, bcdDevice, and bDeviceClass for USB devices
> >> >>   as well as the bInterfaceClass, bInterfaceSubClass, and
> >> >>   bInterfaceProtocol for each interface
> >> >>
> >> >> * The contents of /etc/os-release
> >> >>
> >> >> * All the RPMs installed on a system
> >> >>
> >> >> Other than the drivers bound to the PCI and USB devices (which is an
> >> >> open PR[3]), what else would be good to collect?
> >> >>
> >> >> [0] https://fedoraproject.org/wiki/Smolt_retirement
> >> >> [1] https://github.com/npmccallum/census
> >> >> [2] https://github.com/npmccallum/census/blob/master/client/plugins/
> >> >> [3] https://github.com/npmccallum/census/pull/3
> >> >
> >> > Internally, we have been focusing on using 'lshw' as the tool that 
> >> > provides
> >> > all that info and handles all the arch funkiness (and includes firmware).
> >> > If there is anything missing, we have tried to push upstream to that
> >> > project.
> >> >
> >> > Would that cover a lot of the info you are looking for?
> >> >
> >> > Cheers,
> >> > Don
> >> ___
> >> kernel mailing list -- kernel@lists.fedoraproject.org
> >> To unsubscribe send an email to kernel-le...@lists.fedoraproject.org
> ___
> kernel mailing list -- kernel@lists.fedoraproject.org
> To unsubscribe send an email to kernel-le...@lists.fedoraproject.org
___
kernel mailing list -- kernel@lists.fedoraproject.org
To unsubscribe send an email to kernel-le...@lists.fedoraproject.org

Re: Reviving the hardware census

[Nathaniel McCallum]

I just looked at the code for lshw. The master branch already supports
JSON. We just need them to release it.

On Wed, Nov 8, 2017 at 3:23 PM, Don Zickus  wrote:
> On Wed, Nov 08, 2017 at 03:16:24PM -0500, Nathaniel McCallum wrote:
>> I just played around with lshw a bit. We should totally make it export
>> JSON. We can then submit this directly (as one census plugin).
>
> Yes, that is how we use it to update hardware info internally to our Beaker
> instance. :-)
>
> Cheers,
> Don
>
>>
>> On Wed, Nov 8, 2017 at 12:34 PM, Don Zickus  wrote:
>> > On Tue, Nov 07, 2017 at 10:49:02PM +, Jeremy Cline wrote:
>> >> Hey folks,
>> >>
>> >> For some time now, Fedora has operated without a database of hardware
>> >> users have. Smolt, the old hardware database, was retired in 2012[0] and
>> >> its intended successor[1] was never deployed by Fedora Infrastructure.
>> >>
>> >> It would be nice to have a hardware database, so I (and hopefully some
>> >> others) would like to get Census up and running for Fedora. Before we
>> >> look at deploying Census, however, it would be good to make sure it has
>> >> everything we need.
>> >>
>> >> Census has client plugins to collect information[2]. At the moment, it
>> >> has plugins for:
>> >>
>> >> * The vendor, device, subsystem_vendor, subsystem_device, and class from
>> >>   each PCI device
>> >>
>> >> * The idVendor, idProduct, bcdDevice, and bDeviceClass for USB devices
>> >>   as well as the bInterfaceClass, bInterfaceSubClass, and
>> >>   bInterfaceProtocol for each interface
>> >>
>> >> * The contents of /etc/os-release
>> >>
>> >> * All the RPMs installed on a system
>> >>
>> >> Other than the drivers bound to the PCI and USB devices (which is an
>> >> open PR[3]), what else would be good to collect?
>> >>
>> >> [0] https://fedoraproject.org/wiki/Smolt_retirement
>> >> [1] https://github.com/npmccallum/census
>> >> [2] https://github.com/npmccallum/census/blob/master/client/plugins/
>> >> [3] https://github.com/npmccallum/census/pull/3
>> >
>> > Internally, we have been focusing on using 'lshw' as the tool that provides
>> > all that info and handles all the arch funkiness (and includes firmware).
>> > If there is anything missing, we have tried to push upstream to that
>> > project.
>> >
>> > Would that cover a lot of the info you are looking for?
>> >
>> > Cheers,
>> > Don
>> ___
>> kernel mailing list -- kernel@lists.fedoraproject.org
>> To unsubscribe send an email to kernel-le...@lists.fedoraproject.org
___
kernel mailing list -- kernel@lists.fedoraproject.org
To unsubscribe send an email to kernel-le...@lists.fedoraproject.org

Re: Reviving the hardware census

[Nathaniel McCallum]

External plugins? No. We are talking about internal modular interfaces
used to separate the code conceptually. This allows us to delegate
data collection easily to domain experts. It also allows users to
choose, somewhat coursely, which day they report. For example, some
users may be fine with submitting hardware but not a list of their
installed packages. Dividing the data cleanly enables this.

On Wed, Nov 8, 2017 at 3:59 PM, Jeremy Cline  wrote:
> On 11/08/2017 03:18 PM, Nathaniel McCallum wrote:
>> I agree completely. My point is not that we don't need any planning,
>> but that the planing is scoped per plugin.
>
> Do we really need the concept of plugins, though? Are there going to be
> plugins that live outside of the census "core"? Will users want to mix
> and match plugins? Are all the plugins combined more than most users
> want to install? I don't feel like the answer to any of those questions
> is "yes".
>
> I'm all for defining solid internal interfaces so it's easy to extend,
> and I expect users will want to be able to limit what they send in
> (maybe just hardware and no software report), but neither of those
> things seem like they require plugins.
>
> Perhaps I'm being short-sighted. I wasn't around for Smolt and so it's
> hard for me to know all the pain-points of its design. It just seems to
> me that scoping planning to individual pieces of the system is going to
> lead to a whole bunch of disconnected buckets of data that won't be
> pleasant to work with.
>
>
> --
> Jeremy Cline
> XMPP: jer...@jcline.org
> IRC:  jcline
>
___
kernel mailing list -- kernel@lists.fedoraproject.org
To unsubscribe send an email to kernel-le...@lists.fedoraproject.org

Re: Reviving the hardware census

[Jeremy Cline]

On 11/08/2017 03:18 PM, Nathaniel McCallum wrote:
> I agree completely. My point is not that we don't need any planning,
> but that the planing is scoped per plugin.

Do we really need the concept of plugins, though? Are there going to be
plugins that live outside of the census "core"? Will users want to mix
and match plugins? Are all the plugins combined more than most users
want to install? I don't feel like the answer to any of those questions
is "yes".

I'm all for defining solid internal interfaces so it's easy to extend,
and I expect users will want to be able to limit what they send in
(maybe just hardware and no software report), but neither of those
things seem like they require plugins.

Perhaps I'm being short-sighted. I wasn't around for Smolt and so it's
hard for me to know all the pain-points of its design. It just seems to
me that scoping planning to individual pieces of the system is going to
lead to a whole bunch of disconnected buckets of data that won't be
pleasant to work with.


-- 
Jeremy Cline
XMPP: jer...@jcline.org
IRC:  jcline

___
kernel mailing list -- kernel@lists.fedoraproject.org
To unsubscribe send an email to kernel-le...@lists.fedoraproject.org

Re: Reviving the hardware census

[Don Zickus]

On Wed, Nov 08, 2017 at 03:16:24PM -0500, Nathaniel McCallum wrote:
> I just played around with lshw a bit. We should totally make it export
> JSON. We can then submit this directly (as one census plugin).

Yes, that is how we use it to update hardware info internally to our Beaker
instance. :-)

Cheers,
Don

> 
> On Wed, Nov 8, 2017 at 12:34 PM, Don Zickus  wrote:
> > On Tue, Nov 07, 2017 at 10:49:02PM +, Jeremy Cline wrote:
> >> Hey folks,
> >>
> >> For some time now, Fedora has operated without a database of hardware
> >> users have. Smolt, the old hardware database, was retired in 2012[0] and
> >> its intended successor[1] was never deployed by Fedora Infrastructure.
> >>
> >> It would be nice to have a hardware database, so I (and hopefully some
> >> others) would like to get Census up and running for Fedora. Before we
> >> look at deploying Census, however, it would be good to make sure it has
> >> everything we need.
> >>
> >> Census has client plugins to collect information[2]. At the moment, it
> >> has plugins for:
> >>
> >> * The vendor, device, subsystem_vendor, subsystem_device, and class from
> >>   each PCI device
> >>
> >> * The idVendor, idProduct, bcdDevice, and bDeviceClass for USB devices
> >>   as well as the bInterfaceClass, bInterfaceSubClass, and
> >>   bInterfaceProtocol for each interface
> >>
> >> * The contents of /etc/os-release
> >>
> >> * All the RPMs installed on a system
> >>
> >> Other than the drivers bound to the PCI and USB devices (which is an
> >> open PR[3]), what else would be good to collect?
> >>
> >> [0] https://fedoraproject.org/wiki/Smolt_retirement
> >> [1] https://github.com/npmccallum/census
> >> [2] https://github.com/npmccallum/census/blob/master/client/plugins/
> >> [3] https://github.com/npmccallum/census/pull/3
> >
> > Internally, we have been focusing on using 'lshw' as the tool that provides
> > all that info and handles all the arch funkiness (and includes firmware).
> > If there is anything missing, we have tried to push upstream to that
> > project.
> >
> > Would that cover a lot of the info you are looking for?
> >
> > Cheers,
> > Don
> ___
> kernel mailing list -- kernel@lists.fedoraproject.org
> To unsubscribe send an email to kernel-le...@lists.fedoraproject.org
___
kernel mailing list -- kernel@lists.fedoraproject.org
To unsubscribe send an email to kernel-le...@lists.fedoraproject.org

Re: Reviving the hardware census

[Nathaniel McCallum]

I agree completely. My point is not that we don't need any planning,
but that the planing is scoped per plugin.

On Wed, Nov 8, 2017 at 3:05 PM, Jeremy Cline  wrote:
> On 11/08/2017 09:24 AM, Nathaniel McCallum wrote:
>> Here is why I don't think we need to have all the data collection
>> requirements up front. Clevis is designed to be very modular. A data
>> collector plugin is just an executable that outputs a JSON blob. A
>> corresponding server-side plugin parses this data and stores it in the
>> database in an efficient way to be later queried.
>
> I certainly don't expect to walk away from this with everything everyone
> wants (and nothing people don't want), but it doesn't hurt to spend a
> little time up-front thinking about this.
>
> Since we're going to be putting this in a database and hopefully we have
> a lot of data, thinking about what that data will look like now will
> save us and those who have to maintain this system in production a great
> deal of pain. It looks like that current implementation uses MongoDB,
> which could either be a good choice or a bad choice depending on what
> the data schema looks like. If it is a good choice, we need to be able
> to prove that since Fedora Infrastructure uses PostgreSQL for pretty
> much everything and they probably won't be thrilled about maintaining
> another database.
>
> If we opt for PostgreSQL that means we really do have to model our data
> (I think is a good thing to do even with something like MongoDB) so we
> should have a pretty solid idea of what it looks like.
>
>
> --
> Jeremy Cline
> XMPP: jer...@jcline.org
> IRC:  jcline
>
___
kernel mailing list -- kernel@lists.fedoraproject.org
To unsubscribe send an email to kernel-le...@lists.fedoraproject.org

Re: Reviving the hardware census

[Nathaniel McCallum]

I just played around with lshw a bit. We should totally make it export
JSON. We can then submit this directly (as one census plugin).

On Wed, Nov 8, 2017 at 12:34 PM, Don Zickus  wrote:
> On Tue, Nov 07, 2017 at 10:49:02PM +, Jeremy Cline wrote:
>> Hey folks,
>>
>> For some time now, Fedora has operated without a database of hardware
>> users have. Smolt, the old hardware database, was retired in 2012[0] and
>> its intended successor[1] was never deployed by Fedora Infrastructure.
>>
>> It would be nice to have a hardware database, so I (and hopefully some
>> others) would like to get Census up and running for Fedora. Before we
>> look at deploying Census, however, it would be good to make sure it has
>> everything we need.
>>
>> Census has client plugins to collect information[2]. At the moment, it
>> has plugins for:
>>
>> * The vendor, device, subsystem_vendor, subsystem_device, and class from
>>   each PCI device
>>
>> * The idVendor, idProduct, bcdDevice, and bDeviceClass for USB devices
>>   as well as the bInterfaceClass, bInterfaceSubClass, and
>>   bInterfaceProtocol for each interface
>>
>> * The contents of /etc/os-release
>>
>> * All the RPMs installed on a system
>>
>> Other than the drivers bound to the PCI and USB devices (which is an
>> open PR[3]), what else would be good to collect?
>>
>> [0] https://fedoraproject.org/wiki/Smolt_retirement
>> [1] https://github.com/npmccallum/census
>> [2] https://github.com/npmccallum/census/blob/master/client/plugins/
>> [3] https://github.com/npmccallum/census/pull/3
>
> Internally, we have been focusing on using 'lshw' as the tool that provides
> all that info and handles all the arch funkiness (and includes firmware).
> If there is anything missing, we have tried to push upstream to that
> project.
>
> Would that cover a lot of the info you are looking for?
>
> Cheers,
> Don
___
kernel mailing list -- kernel@lists.fedoraproject.org
To unsubscribe send an email to kernel-le...@lists.fedoraproject.org

Re: Reviving the hardware census

[Jeremy Cline]

On 11/08/2017 09:24 AM, Nathaniel McCallum wrote:
> Here is why I don't think we need to have all the data collection
> requirements up front. Clevis is designed to be very modular. A data
> collector plugin is just an executable that outputs a JSON blob. A
> corresponding server-side plugin parses this data and stores it in the
> database in an efficient way to be later queried.

I certainly don't expect to walk away from this with everything everyone
wants (and nothing people don't want), but it doesn't hurt to spend a
little time up-front thinking about this.

Since we're going to be putting this in a database and hopefully we have
a lot of data, thinking about what that data will look like now will
save us and those who have to maintain this system in production a great
deal of pain. It looks like that current implementation uses MongoDB,
which could either be a good choice or a bad choice depending on what
the data schema looks like. If it is a good choice, we need to be able
to prove that since Fedora Infrastructure uses PostgreSQL for pretty
much everything and they probably won't be thrilled about maintaining
another database.

If we opt for PostgreSQL that means we really do have to model our data
(I think is a good thing to do even with something like MongoDB) so we
should have a pretty solid idea of what it looks like.


-- 
Jeremy Cline
XMPP: jer...@jcline.org
IRC:  jcline

___
kernel mailing list -- kernel@lists.fedoraproject.org
To unsubscribe send an email to kernel-le...@lists.fedoraproject.org

Re: Reviving the hardware census

[Josh Boyer]

On Wed, Nov 8, 2017 at 2:14 PM, Don Zickus  wrote:
> On Wed, Nov 08, 2017 at 01:48:36PM -0500, Josh Boyer wrote:
>> >> [1] https://github.com/npmccallum/census
>> >> [2] https://github.com/npmccallum/census/blob/master/client/plugins/
>> >> [3] https://github.com/npmccallum/census/pull/3
>> >
>> > Internally, we have been focusing on using 'lshw' as the tool that provides
>> > all that info and handles all the arch funkiness (and includes firmware).
>> > If there is anything missing, we have tried to push upstream to that
>> > project.
>> >
>> > Would that cover a lot of the info you are looking for?
>>
>> It sounds like lshw could provide the output for the local system if
>> someone wrote a census plugin for it.  What it doesn't seem to cover
>> at all is the "gather data and send it somewhere" part, right?
>
> I think it covers part of the 'gather data', no? :-)  I had assumed the
> census tool handles the 'send it' somewhere.

Sorry, I phrased that awkwardly.  I meant "gather the data from
multiple computers and send it to a central localtion".  But I think
we're saying the same thing.

> As part of the kernel CI work I am doing internally, we are trying to figure
> out a more universal way of exchange machine info when providing feedback
> that a test or patch broke.  Lots of folks have been using lshw.  This has
> made it easier to write scripts on top of that output compared to various
> custom output.  It isn't perfect, but it seems to do a reasonable job today.

Right.  Adding a census plugin to consume that could build on top of
it even further.

josh
___
kernel mailing list -- kernel@lists.fedoraproject.org
To unsubscribe send an email to kernel-le...@lists.fedoraproject.org

Re: Reviving the hardware census

[Don Zickus]

On Wed, Nov 08, 2017 at 01:48:36PM -0500, Josh Boyer wrote:
> >> [1] https://github.com/npmccallum/census
> >> [2] https://github.com/npmccallum/census/blob/master/client/plugins/
> >> [3] https://github.com/npmccallum/census/pull/3
> >
> > Internally, we have been focusing on using 'lshw' as the tool that provides
> > all that info and handles all the arch funkiness (and includes firmware).
> > If there is anything missing, we have tried to push upstream to that
> > project.
> >
> > Would that cover a lot of the info you are looking for?
> 
> It sounds like lshw could provide the output for the local system if
> someone wrote a census plugin for it.  What it doesn't seem to cover
> at all is the "gather data and send it somewhere" part, right?

I think it covers part of the 'gather data', no? :-)  I had assumed the
census tool handles the 'send it' somewhere.

As part of the kernel CI work I am doing internally, we are trying to figure
out a more universal way of exchange machine info when providing feedback
that a test or patch broke.  Lots of folks have been using lshw.  This has
made it easier to write scripts on top of that output compared to various
custom output.  It isn't perfect, but it seems to do a reasonable job today.


Cheers,
Don

> 
> josh
> ___
> kernel mailing list -- kernel@lists.fedoraproject.org
> To unsubscribe send an email to kernel-le...@lists.fedoraproject.org
___
kernel mailing list -- kernel@lists.fedoraproject.org
To unsubscribe send an email to kernel-le...@lists.fedoraproject.org

Re: Reviving the hardware census

[Josh Boyer]

On Wed, Nov 8, 2017 at 12:34 PM, Don Zickus  wrote:
> On Tue, Nov 07, 2017 at 10:49:02PM +, Jeremy Cline wrote:
>> Hey folks,
>>
>> For some time now, Fedora has operated without a database of hardware
>> users have. Smolt, the old hardware database, was retired in 2012[0] and
>> its intended successor[1] was never deployed by Fedora Infrastructure.
>>
>> It would be nice to have a hardware database, so I (and hopefully some
>> others) would like to get Census up and running for Fedora. Before we
>> look at deploying Census, however, it would be good to make sure it has
>> everything we need.
>>
>> Census has client plugins to collect information[2]. At the moment, it
>> has plugins for:
>>
>> * The vendor, device, subsystem_vendor, subsystem_device, and class from
>>   each PCI device
>>
>> * The idVendor, idProduct, bcdDevice, and bDeviceClass for USB devices
>>   as well as the bInterfaceClass, bInterfaceSubClass, and
>>   bInterfaceProtocol for each interface
>>
>> * The contents of /etc/os-release
>>
>> * All the RPMs installed on a system
>>
>> Other than the drivers bound to the PCI and USB devices (which is an
>> open PR[3]), what else would be good to collect?
>>
>> [0] https://fedoraproject.org/wiki/Smolt_retirement
>> [1] https://github.com/npmccallum/census
>> [2] https://github.com/npmccallum/census/blob/master/client/plugins/
>> [3] https://github.com/npmccallum/census/pull/3
>
> Internally, we have been focusing on using 'lshw' as the tool that provides
> all that info and handles all the arch funkiness (and includes firmware).
> If there is anything missing, we have tried to push upstream to that
> project.
>
> Would that cover a lot of the info you are looking for?

It sounds like lshw could provide the output for the local system if
someone wrote a census plugin for it.  What it doesn't seem to cover
at all is the "gather data and send it somewhere" part, right?

josh
___
kernel mailing list -- kernel@lists.fedoraproject.org
To unsubscribe send an email to kernel-le...@lists.fedoraproject.org

Re: Reviving the hardware census

[Don Zickus]

On Tue, Nov 07, 2017 at 10:49:02PM +, Jeremy Cline wrote:
> Hey folks,
> 
> For some time now, Fedora has operated without a database of hardware
> users have. Smolt, the old hardware database, was retired in 2012[0] and
> its intended successor[1] was never deployed by Fedora Infrastructure.
> 
> It would be nice to have a hardware database, so I (and hopefully some
> others) would like to get Census up and running for Fedora. Before we
> look at deploying Census, however, it would be good to make sure it has
> everything we need.
> 
> Census has client plugins to collect information[2]. At the moment, it
> has plugins for:
> 
> * The vendor, device, subsystem_vendor, subsystem_device, and class from
>   each PCI device
> 
> * The idVendor, idProduct, bcdDevice, and bDeviceClass for USB devices
>   as well as the bInterfaceClass, bInterfaceSubClass, and
>   bInterfaceProtocol for each interface
> 
> * The contents of /etc/os-release
> 
> * All the RPMs installed on a system
> 
> Other than the drivers bound to the PCI and USB devices (which is an
> open PR[3]), what else would be good to collect?
> 
> [0] https://fedoraproject.org/wiki/Smolt_retirement
> [1] https://github.com/npmccallum/census
> [2] https://github.com/npmccallum/census/blob/master/client/plugins/
> [3] https://github.com/npmccallum/census/pull/3

Internally, we have been focusing on using 'lshw' as the tool that provides
all that info and handles all the arch funkiness (and includes firmware).
If there is anything missing, we have tried to push upstream to that
project.

Would that cover a lot of the info you are looking for?

Cheers,
Don
___
kernel mailing list -- kernel@lists.fedoraproject.org
To unsubscribe send an email to kernel-le...@lists.fedoraproject.org

Re: Reviving the hardware census

[Nathaniel McCallum]

I forgot to post the link to the server-side of the pci plugin:

https://github.com/npmccallum/census/blob/master/libs/census/server/plugins/hardware/pci.py

On Wed, Nov 8, 2017 at 9:24 AM, Nathaniel McCallum
 wrote:
> Here is why I don't think we need to have all the data collection
> requirements up front. Clevis is designed to be very modular. A data
> collector plugin is just an executable that outputs a JSON blob. A
> corresponding server-side plugin parses this data and stores it in the
> database in an efficient way to be later queried.
>
> My goal is to stand up the basic service with some initial limited
> data. Once we prove that this works, domain experts can write the
> plugins to gather the data they want to collect. Here's an example to
> show how easy it would be (this is the PCI device plugin):
>
> https://github.com/npmccallum/census/blob/master/client/plugins/hardware.pci
>
> A really great example is the one Peter Robinson just gave. I know
> nothing about "SoC attached" devices. Nor should I. I can just let
> Peter write the client and server plugins and help review them for
> semantic correctness. He can also craft the queries he wants. But
> since he's the domain expert, he get's to make most of the important
> decisions.
>
> If census does its job right, it totally offloads data collection and
> analysis to the people who know what they are doing with that data.
>
>
> On Tue, Nov 7, 2017 at 5:49 PM, Jeremy Cline  wrote:
>> Hey folks,
>>
>> For some time now, Fedora has operated without a database of hardware
>> users have. Smolt, the old hardware database, was retired in 2012[0] and
>> its intended successor[1] was never deployed by Fedora Infrastructure.
>>
>> It would be nice to have a hardware database, so I (and hopefully some
>> others) would like to get Census up and running for Fedora. Before we
>> look at deploying Census, however, it would be good to make sure it has
>> everything we need.
>>
>> Census has client plugins to collect information[2]. At the moment, it
>> has plugins for:
>>
>> * The vendor, device, subsystem_vendor, subsystem_device, and class from
>>   each PCI device
>>
>> * The idVendor, idProduct, bcdDevice, and bDeviceClass for USB devices
>>   as well as the bInterfaceClass, bInterfaceSubClass, and
>>   bInterfaceProtocol for each interface
>>
>> * The contents of /etc/os-release
>>
>> * All the RPMs installed on a system
>>
>> Other than the drivers bound to the PCI and USB devices (which is an
>> open PR[3]), what else would be good to collect?
>>
>> [0] https://fedoraproject.org/wiki/Smolt_retirement
>> [1] https://github.com/npmccallum/census
>> [2] https://github.com/npmccallum/census/blob/master/client/plugins/
>> [3] https://github.com/npmccallum/census/pull/3
>>
>>
>> --
>> Jeremy Cline
>> XMPP: jer...@jcline.org
>> IRC:  jcline
>>
___
kernel mailing list -- kernel@lists.fedoraproject.org
To unsubscribe send an email to kernel-le...@lists.fedoraproject.org

Re: Reviving the hardware census

[Nathaniel McCallum]

Here is why I don't think we need to have all the data collection
requirements up front. Clevis is designed to be very modular. A data
collector plugin is just an executable that outputs a JSON blob. A
corresponding server-side plugin parses this data and stores it in the
database in an efficient way to be later queried.

My goal is to stand up the basic service with some initial limited
data. Once we prove that this works, domain experts can write the
plugins to gather the data they want to collect. Here's an example to
show how easy it would be (this is the PCI device plugin):

https://github.com/npmccallum/census/blob/master/client/plugins/hardware.pci

A really great example is the one Peter Robinson just gave. I know
nothing about "SoC attached" devices. Nor should I. I can just let
Peter write the client and server plugins and help review them for
semantic correctness. He can also craft the queries he wants. But
since he's the domain expert, he get's to make most of the important
decisions.

If census does its job right, it totally offloads data collection and
analysis to the people who know what they are doing with that data.


On Tue, Nov 7, 2017 at 5:49 PM, Jeremy Cline  wrote:
> Hey folks,
>
> For some time now, Fedora has operated without a database of hardware
> users have. Smolt, the old hardware database, was retired in 2012[0] and
> its intended successor[1] was never deployed by Fedora Infrastructure.
>
> It would be nice to have a hardware database, so I (and hopefully some
> others) would like to get Census up and running for Fedora. Before we
> look at deploying Census, however, it would be good to make sure it has
> everything we need.
>
> Census has client plugins to collect information[2]. At the moment, it
> has plugins for:
>
> * The vendor, device, subsystem_vendor, subsystem_device, and class from
>   each PCI device
>
> * The idVendor, idProduct, bcdDevice, and bDeviceClass for USB devices
>   as well as the bInterfaceClass, bInterfaceSubClass, and
>   bInterfaceProtocol for each interface
>
> * The contents of /etc/os-release
>
> * All the RPMs installed on a system
>
> Other than the drivers bound to the PCI and USB devices (which is an
> open PR[3]), what else would be good to collect?
>
> [0] https://fedoraproject.org/wiki/Smolt_retirement
> [1] https://github.com/npmccallum/census
> [2] https://github.com/npmccallum/census/blob/master/client/plugins/
> [3] https://github.com/npmccallum/census/pull/3
>
>
> --
> Jeremy Cline
> XMPP: jer...@jcline.org
> IRC:  jcline
>
___
kernel mailing list -- kernel@lists.fedoraproject.org
To unsubscribe send an email to kernel-le...@lists.fedoraproject.org

Re: Reviving the hardware census

[Peter Robinson]

On Tue, Nov 7, 2017 at 10:49 PM, Jeremy Cline  wrote:
> Hey folks,
>
> For some time now, Fedora has operated without a database of hardware
> users have. Smolt, the old hardware database, was retired in 2012[0] and
> its intended successor[1] was never deployed by Fedora Infrastructure.
>
> It would be nice to have a hardware database, so I (and hopefully some
> others) would like to get Census up and running for Fedora. Before we
> look at deploying Census, however, it would be good to make sure it has
> everything we need.
>
> Census has client plugins to collect information[2]. At the moment, it
> has plugins for:
>
> * The vendor, device, subsystem_vendor, subsystem_device, and class from
>   each PCI device
>
> * The idVendor, idProduct, bcdDevice, and bDeviceClass for USB devices
>   as well as the bInterfaceClass, bInterfaceSubClass, and
>   bInterfaceProtocol for each interface
>
> * The contents of /etc/os-release
>
> * All the RPMs installed on a system
>
> Other than the drivers bound to the PCI and USB devices (which is an
> open PR[3]), what else would be good to collect?

On ARM any platform "SoC attached" would be useful else you'll get
almost nothing as most NICs/SATA/storage/GPU/cameras etc are generally
not attached via USB/PCI. This would also include things like SDIO
(for wifi), and I suppose i2c/gpio would be useful in that context
too.

Peter
___
kernel mailing list -- kernel@lists.fedoraproject.org
To unsubscribe send an email to kernel-le...@lists.fedoraproject.org

Re: Reviving the hardware census

[Bastien Nocera]

- Original Message -
> Hey folks,
> 
> For some time now, Fedora has operated without a database of hardware
> users have. Smolt, the old hardware database, was retired in 2012[0] and
> its intended successor[1] was never deployed by Fedora Infrastructure.
> 
> It would be nice to have a hardware database, so I (and hopefully some
> others) would like to get Census up and running for Fedora. Before we
> look at deploying Census, however, it would be good to make sure it has
> everything we need.
> 
> Census has client plugins to collect information[2]. At the moment, it
> has plugins for:
> 
> * The vendor, device, subsystem_vendor, subsystem_device, and class from
>   each PCI device
> 
> * The idVendor, idProduct, bcdDevice, and bDeviceClass for USB devices
>   as well as the bInterfaceClass, bInterfaceSubClass, and
>   bInterfaceProtocol for each interface
> 
> * The contents of /etc/os-release
> 
> * All the RPMs installed on a system
> 
> Other than the drivers bound to the PCI and USB devices (which is an
> open PR[3]), what else would be good to collect?

I2C devices. PCI and USB would be pretty much everything you'd get on
a desktop or old-school Intel laptop, but for SoC tablets, convertibles
and low-powered laptops, this wouldn't cover much.

The ACPI DSDT would also be useful in figuring out what devices are
unsupported.

> [0] https://fedoraproject.org/wiki/Smolt_retirement
> [1] https://github.com/npmccallum/census
> [2] https://github.com/npmccallum/census/blob/master/client/plugins/
> [3] https://github.com/npmccallum/census/pull/3
> 
> 
> --
> Jeremy Cline
> XMPP: jer...@jcline.org
> IRC:  jcline
> 
> 
> ___
> kernel mailing list -- kernel@lists.fedoraproject.org
> To unsubscribe send an email to kernel-le...@lists.fedoraproject.org
> 
___
kernel mailing list -- kernel@lists.fedoraproject.org
To unsubscribe send an email to kernel-le...@lists.fedoraproject.org