Re: pci_mapreg_map and BAR from `Bus space tutorial'

[Rocky Hotas]

Hi :)!

> Sent: Sunday, June 14, 2020 at 7:21 PM
> From: "Radoslaw Kujawa" 
> To: "Rocky Hotas" 
> Cc: "Martin Husemann" , tech-kern@NetBSD.org
> Subject: Re: pci_mapreg_map and BAR from `Bus space tutorial'

> There's also a good general tutorial about NetBSD drivers, by Jochen
> Kunz, that is referenced at the end of bus_space tutorial.

Yes, I started reading it, too. As regards PCI, it seems sometimes too
wide and slightly off-topic, but it's also a basic and accurate source
of notions.

> The slides were not designed as a stand-alone learning material - I just
> used them to track the progress of a live tutorial, and emphasize
> important areas. So slides obviously lack any background information or
> comments.

I was sure this material was intended to be real-time commented. In fact
some EuroBSDCon video streamings are available on Youtube, but in the
2012 case I could not find your tutorial.
Just in case a video already exists and has been published somewhere, and
only if you want to share it, let me know. But don't worry otherwise.

> As you have probably noticed, the source to slides and examples is
> available on GitHub. However, if you are serious about expanding the
> tutorial please contact me off-list, as I am getting more and more
> hesitant to continue using GitHub for anything (I'll arrange a
> self-hosted repo).

Got it. It's now too early to discuss about this, but if I manage to
write something, I'll keep this in mind and inform you.

> Sure, feel free to contact me.
>
> Due to time constraints, I cannot guarantee a timely response, but I am
> very much interested in any further problems, or general comments that
> you may have.

Thank you so much, I'm glad to know this. No problem for any time-delay.

> I honeslty have no idea, debugging this would require digging into
> gxemul internals.

Ok! This is just a secondary issue now, but I mentioned it FTR.
Bye!

Rocky

Re: pci_mapreg_map and BAR from `Bus space tutorial'

[Radoslaw Kujawa]

Hi.

On 6/12/20 3:30 PM, Rocky Hotas wrote:


I applied your patch and it works for me the same way. Thanks!


Excellent, glad to hear that.


As a sidenote: I found your bus_space_tutorial very useful, the only one
available about this topic. For this problem, I suspended it at slide
47 of your pdf.


There's also a good general tutorial about NetBSD drivers, by Jochen 
Kunz, that is referenced at the end of bus_space tutorial.



My intention is to follow it entirely, and then to elaborate and expend
it (adding comments, citations from the manpages), in order to keep a
useful and up-to-date introduction for a newbie. Would you agree?


Yes, definitely.

Originally, the slides were used as a part of hands-on tutorial 
delivered by me during EuroBSDCon 2012 (and later for some commercial 
clients).


The slides were not designed as a stand-alone learning material - I just 
used them to track the progress of a live tutorial, and emphasize 
important areas. So slides obviously lack any background information or 
comments.


As you have probably noticed, the source to slides and examples is 
available on GitHub. However, if you are serious about expanding the 
tutorial please contact me off-list, as I am getting more and more 
hesitant to continue using GitHub for anything (I'll arrange a 
self-hosted repo).



I hope to be able to follow the remainder of the tutorial by my own. But
if some other problem could arise, can I contact you in future?
Obviously only if and when strictly necessary.


Sure, feel free to contact me.

Due to time constraints, I cannot guarantee a timely response, but I am 
very much interested in any further problems, or general comments that 
you may have.



P.S.
I don't know if it's relevant, but something weird instead happens if I
try to write 0x in the faa BAR.
Before your patch, after the write, the BAR provided the value
0xFFF0: I was expecting 0xFFF0.
After your patch, and after writing 0x, BAR showed an even weirder
value: 0x000F. I used pcictl(8) for these operations. However, it all
seems to work from the dmesg, so maybe they are due to the fact that
this PCI device is not real.


I honeslty have no idea, debugging this would require digging into 
gxemul internals.


Best regards,
Radoslaw

Re: pci_mapreg_map and BAR from `Bus space tutorial'

[Rocky Hotas]

Hello Radoslaw,

On giu 11  1:02, Radoslaw Kujawa wrote:

> It seems that in 2018 cobalt port was switched to common MIPS bus space
> implementation. As a part of this implementation, it started to be more
> strict about verification of addresses used by device during bus_space_map.
[...] 
> Please try the attached gxemul patch.

Thank you so much for this message and the previous ones. Even if I do not
have the necessary knowledge, your comments gave several clarifications.

Also, thanks to Martin for mentioning you and for all his help.

Yes, the 2012 version of gxemul apparently can't compile due to those
warnings treated as errors. So, I tried to figure out what modifications
you made to it (not as many; they can be found through a simple recursive
grep) and applied them to the current gxemul version. This way, I could
use it.

I applied your patch and it works for me the same way. Thanks!

As a sidenote: I found your bus_space_tutorial very useful, the only one
available about this topic. For this problem, I suspended it at slide
47 of your pdf.
My intention is to follow it entirely, and then to elaborate and expend
it (adding comments, citations from the manpages), in order to keep a
useful and up-to-date introduction for a newbie. Would you agree?

I hope to be able to follow the remainder of the tutorial by my own. But
if some other problem could arise, can I contact you in future?
Obviously only if and when strictly necessary. 

In the meanwhile, thanks a lot for your time.

Bye!

Rocky


P.S.
I don't know if it's relevant, but something weird instead happens if I
try to write 0x in the faa BAR.
Before your patch, after the write, the BAR provided the value
0xFFF0: I was expecting 0xFFF0.
After your patch, and after writing 0x, BAR showed an even weirder
value: 0x000F. I used pcictl(8) for these operations. However, it all
seems to work from the dmesg, so maybe they are due to the fact that
this PCI device is not real.

Re: pci_mapreg_map and BAR from `Bus space tutorial'

[Mouse]

>> [in your driver: extern int mapreg_debug; ... mapreg_debug = 1;]
>> int mapreg_debug = 0;
>>  ...
>>  if (mapreg_debug) printf("...", ...);

>> It's a horrible thing to do for "permanent" (production) code, but I
>> see nothing at all wrong with it for experimental debugging.

> Thank you so much!  I am not sure now to have figured it out
> correctly, but I can try.  Does this have any relation with setting
> pci_conf_debug to 1 in `src/sys/dev/pci/pciconf.c', which was
> suggested in another reply?

Only in that each turns on PCI debugging code.  The difference is that
with what I sketched, the debugging code it turns on is code you add,
whereas the debugging code pci_conf_debug turns on is code that's
already there (though code you add can of course be controlled by
anything you want, including pci_conf_debug if you choose).

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: pci_mapreg_map and BAR from `Bus space tutorial'

[Rocky Hotas]

On giu 10  7:05, Mouse wrote:

> You can't.  If you have read-only access to the BAR - which is what
> pcictl gives you - then all you can get is the base address.  You need
> read/write access to get the range as well.  (In case you haven't yet
> seen the trick: you read the value and save it, then you write all-1s
> and read it back, to see which bits are read-only, then you write back
> the value you saved.  Yes, this implies that the size of the range is
> always a power of two.  As I said, clever, but I wish they'd kept their
> cleverness a bit more in check.)

Oh, got it. I checked some more documentation and tried the procedure
you mentioned. It works as expected: for example, my WiFi card ral0
requires a 2^16 bytes range, in its only active BAR. Great!

> I don't.  I have never worked with a cobalt and I don't know whether
> there is a de-facto standard for how MIPS interfaces to PCI.

NP. But if you know some source of documentation about x86 or some other
architectures, it is useful as well.

> Can't you print things?  I've done that often enough with kernel code.
> If -current has broken "printf from the driver" debugging, I'd call
> that a crippling regression.
[...]
> You could always do something like
> 
> (in your driver's .c file)
[...] 
> (in the implementation of pci_mapreg_map)
> 
> int mapreg_debug = 0;
>   ...
>   if (mapreg_debug) printf("...", ...);
> 
> It's a horrible thing to do for "permanent" (production) code, but I
> see nothing at all wrong with it for experimental debugging.

Thank you so much! I am not sure now to have figured it out correctly,
but I can try. Does this have any relation with setting pci_conf_debug
to 1 in `src/sys/dev/pci/pciconf.c', which was suggested in another
reply?

Rocky

Re: pci_mapreg_map and BAR from `Bus space tutorial'

[Radoslaw Kujawa]

On 6/10/20 4:30 PM, Radoslaw Kujawa wrote:


It is more likely that something mis-compiles now (I see a ton of 
warnings trying to build it now), or that there is a bug in 
implementation of test device emulation, that somehow went unnoticed 
back in 2012.


It seems that in 2018 cobalt port was switched to common MIPS bus space 
implementation. As a part of this implementation, it started to be more 
strict about verification of addresses used by device during bus_space_map.


I earlier wrote that gxemul does not emulate Cobalt firmware and that is 
true, however it seems that it does place PCI device at certain 
addresses (0x1011) upon boot.


NetBSD attempts to reconfigure the device to use address 0x1200. As 
a part of switch to common MIPS bus space, for cobalt, window 
0x1200-0x1400 was dedicated to PCI memory BARs.


For reasons still not researched thoroughly, this reconfiguration fails, 
and the device stays at original address.


So when the pci_mapreg_map is called, a MIPS MI bus_space_map is 
executed. As a part of this, translation takes place, however sanity 
check in __BS(translate) throws EINVAL, due to device being outside of 
window range.


I've worked the problem around by placing the device in gxemul at an 
exact address expected by NetBSD:


[   1.000] faa0 at pci0 dev 12 function 0: vendor fabc product 0001 
(rev. 0x01)

[   1.000] faa0: registers at 0x1200
[   1.000] faa0: just checking: 1 + 2 = 3

Please try the attached gxemul patch.

Best regards,
Radoslaw
diff --git a/gxemul-current/src/devices/bus_pci.cc b/gxemul-current/src/devices/bus_pci.cc
index ce64aa0..8e7827c 100644
--- a/gxemul-current/src/devices/bus_pci.cc
+++ b/gxemul-current/src/devices/bus_pci.cc
@@ -1436,6 +1436,7 @@ PCIINIT(ati_radeon_9200_2)
 PCIINIT(faa)
 {
uint64_t port, memaddr;
+   memaddr = 0x1200;
char tmpstr[200];
 
PCI_SET_DATA(PCI_ID_REG, PCI_ID_CODE(PCI_VENDOR_FAKECARDS,
@@ -1444,7 +1445,9 @@ PCIINIT(faa)
PCI_SET_DATA(PCI_CLASS_REG, PCI_CLASS_CODE(PCI_CLASS_PROCESSOR,
PCI_SUBCLASS_PROCESSOR_COPROC, 0) + 0x01);
 
-   allocate_device_space(pd, 0x0, 0x100, &port, &memaddr);
+   PCI_SET_DATA(0x10, memaddr);
+
+// allocate_device_space(pd, 0x0, 0x100, &port, &memaddr);
 
snprintf(tmpstr, sizeof(tmpstr), "faa addr=0x%llx "
"pci_little_endian=1", (long long)memaddr);

Re: pci_mapreg_map and BAR from `Bus space tutorial'

[Radoslaw Kujawa]

On 6/10/20 4:19 PM, Martin Husemann wrote:

On Wed, Jun 10, 2020 at 04:12:03PM +0200, Radoslaw Kujawa wrote:

I suspect there are two options: either something has bit-rotted in
Cobalt-specific PCI code, or the ancient gxemul from 2012 does not work
correctly anymore.


FWIW, Anders Gavare is quite responsive to email, if there is a hint this
would be a bug in gxemul (but it does not really sound that way right now).

The gxemul for tutorial purposes (with the test device added), was 
forked in 2012.


It is more likely that something mis-compiles now (I see a ton of 
warnings trying to build it now), or that there is a bug in 
implementation of test device emulation, that somehow went unnoticed 
back in 2012.


Best regards,
Radoslaw

Re: pci_mapreg_map and BAR from `Bus space tutorial'

[Martin Husemann]

On Wed, Jun 10, 2020 at 04:12:03PM +0200, Radoslaw Kujawa wrote:
> I suspect there are two options: either something has bit-rotted in
> Cobalt-specific PCI code, or the ancient gxemul from 2012 does not work
> correctly anymore.

FWIW, Anders Gavare is quite responsive to email, if there is a hint this
would be a bug in gxemul (but it does not really sound that way right now).

Martin

Re: pci_mapreg_map and BAR from `Bus space tutorial'

[Radoslaw Kujawa]

On 6/10/20 12:58 PM, Rocky Hotas wrote:

On giu 10 12:54, Rocky Hotas wrote:


the address 0x1011 belongs to a `user process virtual space, mapped
and cached', both when the processor uses 32 and 64 bit addresses.




The address reported by pcictl is a physical address. It looks correct 
in my opinion.


NetBSD is mapping PCI devices on cobalt starting at address 0x1000. 
This is defined in src/sys/arch/cobalt/include/cpu.h.


gxemul does not really emulate cobalt firmware, so the whole setup of 
PCI bus is done via PCI_NETBSD_CONFIGURE mechanism. Considering that 
device was mapped to a sane looking address, I can guess this part has 
worked successfully.


If you want to double check this, I suggest editing 
src/sys/dev/pci/pciconf.c and setting pci_conf_debug to 1. This should 
give you detailed output of configuration process.


The real question boils down to why pci_mapreg_map() fails and I am 
afraid, this requires inserting a few printfs into this function, as 
suggested by other people on the mailing list.


I suspect there are two options: either something has bit-rotted in 
Cobalt-specific PCI code, or the ancient gxemul from 2012 does not work 
correctly anymore.


Nevertheless, this requires further debugging.

Best regards,
Radoslaw

Re: pci_mapreg_map and BAR from `Bus space tutorial'

[Mouse]

>> Guessing should not be relevant; how BARs work is part of the PCI
>> definition.
> What I was meaning is that it seems counterintuitive that a single 32
> bit register can at the same time specify a 32 bit physical memory
> address and a memory region extent.

Yes.  It is clever - but I wish the PCI designers had chosen a bit less
cleverness.

> In the example of the output of `pcictl' for my test device, this is
> exactly the case: only 1 non-zero BAR,

> Base address register at 0x10
>   type: 32-bit nonprefetchable memory
>   base: 0x1011

> I am stuck in trying to interpret this value, and to use it both for
> an address and a range.

You can't.  If you have read-only access to the BAR - which is what
pcictl gives you - then all you can get is the base address.  You need
read/write access to get the range as well.  (In case you haven't yet
seen the trick: you read the value and save it, then you write all-1s
and read it back, to see which bits are read-only, then you write back
the value you saved.  Yes, this implies that the size of the range is
always a power of two.  As I said, clever, but I wish they'd kept their
cleverness a bit more in check.)

> Yes, this may change according to the architecture.  Here it's not
> x86.  If you remember some source, or documentation, I would check it
> out.

I don't.  I have never worked with a cobalt and I don't know whether
there is a de-facto standard for how MIPS interfaces to PCI.

> This makes me wonder if the tutorial is correct and why.

> The main fact here (at least, for a beginner) is that it's extremely
> difficult to make multiple attempts, check and verify, as you would
> with a normal C program (print something, do anything useful to let
> you know how it works).

Can't you print things?  I've done that often enough with kernel code.
If -current has broken "printf from the driver" debugging, I'd call
that a crippling regression.

> Here, for example, I wonder what (in detail) makes pci_mapreg_map
> fail, what kind of space actually needs the PCI test device: in other
> words, some more information about this mapping.

You could always do something like

(in your driver's .c file)

extern int mapreg_debug;

..._attach(...)
{
...
mapreg_debug = 1;
... pci_mapreg_map(...) ...
mapreg_debug = 0;
...
}

(in the implementation of pci_mapreg_map)

int mapreg_debug = 0;
...
if (mapreg_debug) printf("...", ...);

It's a horrible thing to do for "permanent" (production) code, but I
see nothing at all wrong with it for experimental debugging.

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: pci_mapreg_map and BAR from `Bus space tutorial'

[Rocky Hotas]

On giu 10 12:54, Rocky Hotas wrote:

> the address 0x1011 belongs to a `user process virtual space, mapped
> and cached', both when the processor uses 32 and 64 bit addresses.

No, sorry, that was about Virtual addresses, not physical ones. Here,

 


it is only stated that ``the relationship between the PCI bus address
space and the system memory physical address space differs from one
system type to another''.
So, basically it gives no information.

Rocky

Re: pci_mapreg_map and BAR from `Bus space tutorial'

[Rocky Hotas]

On giu 10 11:50, Martin Husemann wrote:

> so it is PCI_MAPREG_TYPE_MEM ("memory" in the type line), and the mapping
> should just work.

Ok! And I guess that invoking pci_mapreg_type() would give the same
result.

> Does the map address (0x1011) make sense on that machine?

I really don't know. The manual of MIPS R4000 is available at

 

but not the one of R5000, the processory family which should be used in the
emulated Cobalt machine:

 

Here,

 


(``an overview of the management of physical and virtual memory in the
MIPS R4x00, R5000, R8000, and R1 processors'')

the address 0x1011 belongs to a `user process virtual space, mapped
and cached', both when the processor uses 32 and 64 bit addresses. I
think however that the emulated Cobalt in gxemul uses 32 bit addresses.

Rocky

Re: pci_mapreg_map and BAR from `Bus space tutorial'

[Martin Husemann]

On Wed, Jun 10, 2020 at 11:41:51AM +0200, Rocky Hotas wrote:
> Base address register at 0x10
>   type: 32-bit nonprefetchable memory
>   base: 0x1011

so it is PCI_MAPREG_TYPE_MEM ("memory" in the type line), and the mapping
should just work.

Does the map address (0x1011) make sense on that machine?

Martin

Re: pci_mapreg_map and BAR from `Bus space tutorial'

[Rocky Hotas]

Hi!

On giu 08 15:31, Mouse wrote:
> Guessing should not be relevant; how BARs work is part of the PCI
> definition.

Yes, of course. What I was meaning is that it seems counterintuitive
that a single 32 bit register can at the same time specify a 32 bit
physical memory address and a memory region extent.
In the example of the output of `pcictl' for my test device, this is
exactly the case: only 1 non-zero BAR,

Base address register at 0x10
  type: 32-bit nonprefetchable memory
  base: 0x1011

I am stuck in trying to interpret this value, and to use it both for an
address and a range.

> I forget details, but I think things like "I/O mapping
> versus memory mapping" are present in read-only bits at the LSB end of
> the BAR.  NetBSD has a bunch of #defines for them; look for PCI_MAPREG_
> in pcireg.h - or, at least, that's where they are on 1.4T, 5.2, and
> 8.0; while I don't have anything more recent at ready hand to check,
> I'd guess that something that stable is probably going to stay stable.

It is still as you said, IIUC: starting from line 453 of the current
revision 1.138.2.2,

#define PCI_MAPREG_TYPE_MEM 0x
#define PCI_MAPREG_TYPE_ROM 0x
#define PCI_MAPREG_TYPE_IO  0x0001
#define PCI_MAPREG_ROM_ENABLE   0x0001

> >> Give this is mips I assume PCI_MAPREG_TYPE_MEM is correct, but for a
> >> new device you may be unsure, and sometimes variants of devices
> >> exist that either make the type IO or MEM.  You can query that in
> >> your driver with pci_mapreg_info() - but for the concrete case the
> >> userland pcictl output is easier.
> 
> In this particular case, sure; I suspect the person who wrote the
> double-quoted text above was trying to give advice applicable beyond
> the case at immediate hand.

Got it!

> PCI is defined to have two address spaces, called I/O space and memory
> space.  On x86, these normally correspond to I/O instructions
> (inb/outb/etc) and memory-mapped I/O, respectively.  On other machines,
> this often works differently; some PCI attachments present PCI I/O
> space as memory-mapped too, just in a different part of the address
> space.

Yes, this may change according to the architecture. Here it's not x86.
If you remember some source, or documentation, I would check it out.

> The line you cite
> 
> > Command register: 0x0003
> >   I/O space accesses: on
> 
> is a PCI thing, indicating that the device is configured to respond to
> a PCI access marked as an I/O space access, provided it's mapped by a
> suitable mapping register.

This makes me wonder if the tutorial is correct and why.

The main fact here (at least, for a beginner) is that it's extremely
difficult to make multiple attempts, check and verify, as you would with
a normal C program (print something, do anything useful to let you know
how it works). Again, if you (or someone else) have any advice on how to
do this with kernel dmesg and an example PCI device, it would be very
useful.

Here, for example, I wonder what (in detail) makes pci_mapreg_map fail,
what kind of space actually needs the PCI test device: in other words,
some more information about this mapping.

> For a machine that already has a NetBSD port supporting PCI, those
> details should already be handled for you by the bus_space
> implementation for that port.  Your driver just needs to know "is this
> a (PCI) I/O space mapping or a (PCI) memory space mapping?", and
> sometimes not even that; what it turns into at the machine-code level
> is hidden by the bus_space layer.  See pci_mapreg_type(),
> pci_mapreg_map(), etc.

Maybe I can check the result of pci_mapreg_type(), to obtain more
information.

Anyone who can provide some other basic hints is very welcomed. In the
meanwhile, thanks a lot!

Rocky

Re: pci_mapreg_map and BAR from `Bus space tutorial'

[Mouse]

>>> - FAA_MMREG_BAR may contain the beginning address of a space to be
>>>mapped. But how does `pci_mapreg_map' know the extension of such
>>>a map?
>> [...], the BAR contains all info needed (or should, if setup has
>> worked correctly).
> Oh, ok, so the BAR does not specify just an address: it also
> specifies the extension of the memory space.  I could not guess this.

Guessing should not be relevant; how BARs work is part of the PCI
definition.  I forget details, but I think things like "I/O mapping
versus memory mapping" are present in read-only bits at the LSB end of
the BAR.  NetBSD has a bunch of #defines for them; look for PCI_MAPREG_
in pcireg.h - or, at least, that's where they are on 1.4T, 5.2, and
8.0; while I don't have anything more recent at ready hand to check,
I'd guess that something that stable is probably going to stay stable.

>> Give this is mips I assume PCI_MAPREG_TYPE_MEM is correct, but for a
>> new device you may be unsure, and sometimes variants of devices
>> exist that either make the type IO or MEM.  You can query that in
>> your driver with pci_mapreg_info() - but for the concrete case the
>> userland pcictl output is easier.
> Consider that this is a custom PCI device, created by the author of
> the tutorial as statement in the sourcecode of gxemul [...]

In this particular case, sure; I suspect the person who wrote the
double-quoted text above was trying to give advice applicable beyond
the case at immediate hand.

> Looking into the output of pcictl(8), also `I/O space accesses: on'
> is on, but I don't know if it is significant.

PCI is defined to have two address spaces, called I/O space and memory
space.  On x86, these normally correspond to I/O instructions
(inb/outb/etc) and memory-mapped I/O, respectively.  On other machines,
this often works differently; some PCI attachments present PCI I/O
space as memory-mapped too, just in a different part of the address
space.  (I daresay this could be done on x86 too, if a chipset/board
maker wanted to, but it would render the hardware incompatible with a
lot of OS code, so it's not normally done.)

The line you cite

> Command register: 0x0003
>   I/O space accesses: on

is a PCI thing, indicating that the device is configured to respond to
a PCI access marked as an I/O space access, provided it's mapped by a
suitable mapping register.  If the hardware is something that doesn't
have separate I/O space instructions, then PCI I/O space probably turns
into something else from the CPU's point of view, like a different part
of the address space, or some other way of marking accesses as being in
a different address space (like SPARC's lda and sta instructions, which
take an address-space identifier as well as an address).

For a machine that already has a NetBSD port supporting PCI, those
details should already be handled for you by the bus_space
implementation for that port.  Your driver just needs to know "is this
a (PCI) I/O space mapping or a (PCI) memory space mapping?", and
sometimes not even that; what it turns into at the machine-code level
is hidden by the bus_space layer.  See pci_mapreg_type(),
pci_mapreg_map(), etc.

/~\ The ASCII Mouse
\ / Ribbon Campaign
 X  Against HTMLmo...@rodents-montreal.org
/ \ Email!   7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

Re: pci_mapreg_map and BAR from `Bus space tutorial'

[Rocky Hotas]

Hello Martin,
sorry for the delay and thanks a lot for your answer.

On giu 02  7:03, Martin Husemann wrote:
 
> > - FAA_MMREG_BAR may contain the beginning address of a space to be
> >   mapped. But how does `pci_mapreg_map' know the extension of such a map?
> 
> It is just an offset (like a register number), the BAR contains all info
> needed (or should, if setup has worked correctly).

Oh, ok, so the BAR does not specify just an address: it also specifies
the extension of the memory space. I could not guess this.

> If you remove your driver you can list the details from userland with
> pcictl(8).
[...]
> Usually firmware (or early bootloaders) configure the BARs properly, but
> sometimes they don't, and then NetBSD needs to run fixup code (which is
> a kernel option and not included in most kernels). The output
> of pcictl would help here.

Ok! With or without the driver, the output of pcictl(8) is the same and
I attach it to this message. The device we are talking about is in bus
pci0, dev 12, fun 0, so I ran from a root prompt `pcictl pci0 dump -d 12'.

> Give this is mips I assume PCI_MAPREG_TYPE_MEM is correct, but for a new
> device you may be unsure, and sometimes variants of devices exist that
> either make the type IO or MEM. You can query that in your driver with
> pci_mapreg_info() - but for the concrete case the userland pcictl output is
> easier.

Consider that this is a custom PCI device, created by the author of the
tutorial as statement in the sourcecode of gxemul which defines the
`cobalt' machine.
The tutorial used PCI_MAPREG_TYPE_MEM. Looking into the output of pcictl(8),
also `I/O space accesses: on' is on, but I don't know if it is significant.

Bye!

Rocky
PCI configuration registers:
  Common header:
0x00: 0x0001fabc 0x0003 0x0b41 0x

Vendor ID: 0xfabc
Device ID: 0x0001
Command register: 0x0003
  I/O space accesses: on
  Memory space accesses: on
  Bus mastering: off
  Special cycles: off
  MWI transactions: off
  Palette snooping: off
  Parity error checking: off
  Address/data stepping: off
  System error (SERR): off
  Fast back-to-back transactions: off
  Interrupt disable: off
Status register: 0x
  Immediate Readiness: off
  Interrupt status: inactive
  Capability List support: off
  66 MHz capable: off
  User Definable Features (UDF) support: off
  Fast back-to-back capable: off
  Data parity error detected: off
  DEVSEL timing: fast (0x0)
  Slave signaled Target Abort: off
  Master received Target Abort: off
  Master received Master Abort: off
  Asserted System Error (SERR): off
  Parity error detected: off
Class Name: processor (0x0b)
Subclass Name: Co-processor (0x40)
Interface: 0x00
Revision ID: 0x01
BIST: 0x00
Header Type: 0x00 (0x00)
Latency Timer: 0x00
Cache Line Size: 0bytes (0x00)

  Type 0 ("normal" device) header:
0x10: 0x1011 0x 0x 0x
0x20: 0x 0x 0x 0x
0x30: 0x 0x 0x 0x

Base address register at 0x10
  type: 32-bit nonprefetchable memory
  base: 0x1011
Base address register at 0x14
  not implemented
Base address register at 0x18
  not implemented
Base address register at 0x1c
  not implemented
Base address register at 0x20
  not implemented
Base address register at 0x24
  not implemented
Cardbus CIS Pointer: 0x
Subsystem vendor ID: 0x
Subsystem ID: 0x
Expansion ROM Base Address Register: 0x
  base: 0x
  Expansion ROM Enable: off
  Validation Status: Validation not supported
  Validation Details: 0x0
Reserved @ 0x34: 0x
Reserved @ 0x38: 0x
Maximum Latency: 0x00
Minimum Grant: 0x00
Interrupt pin: 0x00 (none)
Interrupt line: 0x00

  Device-dependent header:
0x40: 0x 0x 0x 0x
0x50: 0x 0x 0x 0x
0x60: 0x 0x 0x 0x
0x70: 0x 0x 0x 0x
0x80: 0x 0x 0x 0x
0x90: 0x 0x 0x 0x
0xa0: 0x 0x 0x 0x
0xb0: 0x 0x 0x 0x
0xc0: 0x 0x 0x 0x
0xd0: 0x 0x 0x 0x
0xe0: 0x 0x 0x 0x
0xf0: 0x 0x 0x 0x

Re: pci_mapreg_map and BAR from `Bus space tutorial'

[Martin Husemann]

On Mon, Jun 01, 2020 at 08:35:13PM +0200, Rocky Hotas wrote:
> pci_aprint_devinfo(pa, NULL);
> 
> if (pci_mapreg_map(pa, FAA_MMREG_BAR, PCI_MAPREG_TYPE_MEM, 0, 
> &sc->sc_regt, &sc->sc_regh, &sc->sc_reg_pa, 0) != 0 ) {
> aprint_error_dev(sc->sc_dev, "can't map the BAR\n");
> return;
> }
> 
> aprint_normal_dev(sc->sc_dev, "regs at 0x%08x\n", (uint32_t)
> sc->sc_reg_pa);
> }
> 
> IIUC from the slides and the manpages, `pci_mapreg_map' (PCI-specific
> way of mapping a memory space) invokes `bus_space_map' (generic
> bus_space way of mapping a memory space).
> 
> My doubts are:
> 
> - What does exactly do `pci_mapreg_map'? Does it map the physical address
>   contained in FAA_MMREG_BAR to a kernel virtual address?

Yes (and often creates a new MMU/pmap mapping for this)

> - FAA_MMREG_BAR may contain the beginning address of a space to be
>   mapped. But how does `pci_mapreg_map' know the extension of such a map?

It is just an offset (like a register number), the BAR contains all info
needed (or should, if setup has worked correctly).

If you remove your driver you can list the details from userland with
pcictl(8).

> - According to pci(9), the ``physical address of the mapping is in''
>   FAA_MMREG_BAR. But this assumes that someone previously wrote
>   FAA_MMREG_BAR with the physical address assigned to that specific PCI
>   device. As regards a NetBSD system, who knows that physical address
>   and who is supposed to write it on this PCI device's BAR, before it
>   can be used in `pci_mapreg_map'? That is: does the OS rely on the BIOS
>   configuration, or does the OS configure the BARs by itself?

Usually firmware (or early bootloaders) configure the BARs properly, but
sometimes they don't, and then NetBSD needs to run fixup code (which is
a kernel option and not included in most kernels). The output
of pcictl would help here.

Give this is mips I assume PCI_MAPREG_TYPE_MEM is correct, but for a new
device you may be unsure, and sometimes variants of devices exist that
either make the type IO or MEM. You can query that in your driver with
pci_mapreg_info() - but for the concrete case the userland pcictl output is
easier.

Martin

pci_mapreg_map and BAR from `Bus space tutorial'

[Rocky Hotas]

Hello!
A very useful tutorial is available for newbies like me in

 
 

with a first introduction to kernel PCI drivers. I hope this is the
right ML to write to, despite the (possibly) elementary questions that
follow.

I created the driver according to all the instructions till slide 47,
but `dmesg' prints: `can't map the BAR'. Being it a test device
inside a Cobalt machine emulated by gxemul, I really am not able to
identify the problem.

The PCI device driver uses the file `src/sys/dev/pci/faareg.h' to define
the BAR position:

#ifndef FAAREG_H
#define FAAREG_H

#define FAA_MMREG_BAR   0x10
 
#endif

(I built the kernel this way, but am not sure about the position of
`#endif').

Then, the `attach' function in the driver main file
`src/sys/dev/pci/faa.c' acts this way:

static void
faa_attach(device_t parent, device_t self, void *aux)
{
struct faa_softc *sc = device_private(self);
const struct pci_attach_args *pa = aux;

sc->sc_dev = self;

pci_aprint_devinfo(pa, NULL);

if (pci_mapreg_map(pa, FAA_MMREG_BAR, PCI_MAPREG_TYPE_MEM, 0, 
&sc->sc_regt, &sc->sc_regh, &sc->sc_reg_pa, 0) != 0 ) {
aprint_error_dev(sc->sc_dev, "can't map the BAR\n");
return;
}

aprint_normal_dev(sc->sc_dev, "regs at 0x%08x\n", (uint32_t)
sc->sc_reg_pa);
}

IIUC from the slides and the manpages, `pci_mapreg_map' (PCI-specific
way of mapping a memory space) invokes `bus_space_map' (generic
bus_space way of mapping a memory space).

My doubts are:

- What does exactly do `pci_mapreg_map'? Does it map the physical address
  contained in FAA_MMREG_BAR to a kernel virtual address?

- FAA_MMREG_BAR may contain the beginning address of a space to be
  mapped. But how does `pci_mapreg_map' know the extension of such a map?

- According to pci(9), the ``physical address of the mapping is in''
  FAA_MMREG_BAR. But this assumes that someone previously wrote
  FAA_MMREG_BAR with the physical address assigned to that specific PCI
  device. As regards a NetBSD system, who knows that physical address
  and who is supposed to write it on this PCI device's BAR, before it
  can be used in `pci_mapreg_map'? That is: does the OS rely on the BIOS
  configuration, or does the OS configure the BARs by itself?

It's not always simple to gather some introductory information and I'm
sorry if my questions are trivial or somewhat naive. If anyone knows
about some documentation (not included in the final references of the
mentioned tutorial), I would check it out.
As well, if anyone knows another simple way to emulate a basic PCI
device for test/educational purposes on NetBSD, it's definitely welcome.
Thank you even just for having read.

Rocky