Re: 5.1-rc1: mpt init crash in scsi_map_dma, dma_4v_map_sg on sparc64

[Robin Murphy]

On 02/04/2019 23:39, Rob Gardner wrote:
On 4/2/19 2:30 PM, Meelis Roos wrote:
[   17.566584] scsi host0: ioc0: LSISAS1064 A3, FwRev=010a0000h, 
Ports=1, MaxQ=511, IRQ=27
[   17.595897] mptsas: ioc0: attaching ssp device: fw_channel 0, 
fw_id 0, phy 0, sas_addr 0x5000c5001799a45d
[   17.598465] Unable to handle kernel NULL pointer dereference
[   17.598623] tsk->{mm,active_mm}->context = 0000000000000000
[   17.598723] tsk->{mm,active_mm}->pgd = ffff800008802000
[   17.598774]               \|/ ____ \|/
[   17.598774]               "@'/ .. \`@"
[   17.598774]               /_| \__/ |_\
[   17.598774]                  \__U_/
[   17.598894] swapper/0(1): Oops [#1]
[   17.598937] CPU: 12 PID: 1 Comm: swapper/0 Not tainted 5.1.0-rc1 
#118
[   17.598994] TSTATE: 0000000080e01601 TPC: 00000000004483a8 TNPC: 
00000000004483ac Y: 00000000    Not tainted
[   17.599086] TPC: <dma_4v_map_sg+0xe8/0x5c0>

You may use gdb to figure out what the NULL pointer points to:

gdb vmlinux
l *(dma_4v_map_sg+0xe8)

gdb did not parse the file but objdump --disassemble worked and +0xe8 
seems to be 4483a8


Of course that was right there in the panic message, as TPC is the 
address of the instruction that faulted:

ldx  [ %i4 ], %g1

For anyone wishing to dig into this further, here is my off the cuff 
analysis:

I believe the fault is happening on this line:

         base_shift = tbl->table_map_base >> IO_PAGE_SHIFT;

The tbl variable is assigned to one of two values in the statement 
above, but since the register dump shows the value in %i4 was 0x10, that 
strongly suggests that it executed this:

     tbl = &atu->tbl;

Because the offset of the tbl field in struct atu is 0x10, and that was 
computed here:

448384:       b8 07 60 10     add  %i5, 0x10, %i4

(The offset of tbl in struct iommu is 0, so we would have seen that 0 in 
%i4 if it had taken the iommu path.)

 From the register dump, the value in %i5 was 0. And that came from this 
instruction:

4482f4:       fa 58 e2 58     ldx  [ %g3 + 0x258 ], %i5

Likewise, %g3 came from here:

4482d4:       c6 5e 22 18     ldx  [ %i0 + 0x218 ], %g3

And %i0 is arg0, struct device *dev. So the code is loading some field 
in struct device at offset 0x218, which is consistent with the source:

iommu = dev->archdata.iommu;

So %g3 points to struct iommu, and the code is trying to load the value 
at offset 0x258 in that structure, probably this:

atu = iommu->atu;

And atu is the NULL pointer.

Now whether this is the problem, I don't know. It may be that mask 
(*dev->dma_mask) was wrong, causing the code to take the &atu->tbl path 
instead of the &iommu->tbl path. We can see from the code that mask is 
in %g7, and the register dump shows the value of %g7 is fffffffffffffff, 
while DMA_BIT_MASK(32) is in %g1 and is 00000000ffffffff, so this might 
be the result of some confusion over 32 bit vs 64 bit stuff.

Nice deduction! If it was AArch64 asm I might have tried, but I've never 
even seen SPARC asm before :)

FWIW, scripts/faddr2line is your friend when deciphering stacktrace symbols.

In terms of the crash itself, I'd note that there's also been ongoing 
cleanup to fix the remaining places where the DMA API was called with 
NULL instead of the appropriate device - it could be that as a result of 
that, the driver/subsystem here is now taking a path that has not been 
properly exercised before, and/or that it's not quite the right device 
pointer being picked up.

I hope these bits of information help somebody debug further.

Thanks,
Robin.



Rob



00000000004482c0 <dma_4v_map_sg>:
  4482c0:       9d e3 be b0     save  %sp, -336, %sp
  4482c4:       80 a6 e0 03     cmp  %i3, 3
  4482c8:       02 40 00 c1     be,pn   %icc, 4485cc 
<dma_4v_map_sg+0x30c>
  4482cc:       92 10 21 e2     mov  0x1e2, %o1
  4482d0:       80 a0 00 1a     cmp  %g0, %i2
  4482d4:       c6 5e 22 18     ldx  [ %i0 + 0x218 ], %g3
  4482d8:       82 10 20 00     clr  %g1
  4482dc:       84 60 3f ff     subc  %g0, -1, %g2
  4482e0:       83 78 e4 01     movre  %g3, 1, %g1
  4482e4:       80 90 80 01     orcc  %g2, %g1, %g0
  4482e8:       12 40 00 bd     bne,pn   %icc, 4485dc 
<dma_4v_map_sg+0x31c>
  4482ec:       80 a6 e0 01     cmp  %i3, 1
  4482f0:       84 10 20 03     mov  3, %g2
  4482f4:       fa 58 e2 58     ldx  [ %g3 + 0x258 ], %i5
  4482f8:       85 64 60 01     move  %icc, 1, %g2
  4482fc:       b8 0f 20 02     and  %i4, 2, %i4
  448300:       c0 77 a7 f7     clrx  [ %fp + 0x7f7 ]
  448304:       82 10 a0 04     or  %g2, 4, %g1
  448308:       c0 26 60 18     clr  [ %i1 + 0x18 ]
  44830c:       85 7f 14 01     movrne  %i4, %g1, %g2
  448310:       8f 52 00 00     rdpr  %pil, %g7
  448314:       82 11 e0 0e     or  %g7, 0xe, %g1
  448318:       91 90 60 00     wrpr  %g1, 0, %pil
  44831c:       ce 77 a7 bf     stx  %g7, [ %fp + 0x7bf ]
  448320:       0f 00 02 00     sethi  %hi(0x80000), %g7
  448324:       27 00 00 40     sethi  %hi(0x10000), %l3
  448328:       ce 77 a7 df     stx  %g7, [ %fp + 0x7df ]
  44832c:       0f 00 28 21     sethi  %hi(0xa08400), %g7
  448330:       8e 11 e2 b0     or  %g7, 0x2b0, %g7     ! a086b0 
<iommu_batch>
  448334:       f0 71 c0 05     stx  %i0, [ %g7 + %g5 ]
  448338:       82 01 c0 05     add  %g7, %g5, %g1
  44833c:       c4 70 60 08     stx  %g2, [ %g1 + 8 ]
  448340:       84 10 3f ff     mov  -1, %g2
  448344:       c0 70 60 20     clrx  [ %g1 + 0x20 ]
  448348:       c4 70 60 10     stx  %g2, [ %g1 + 0x10 ]
  44834c:       c2 5e 22 00     ldx  [ %i0 + 0x200 ], %g1
  448350:       22 c0 40 0d     brz,a,pn   %g1, 448384 
<dma_4v_map_sg+0xc4>
  448354:       c2 5e 21 e0     ldx  [ %i0 + 0x1e0 ], %g1
  448358:       e6 00 40 00     ld  [ %g1 ], %l3
  44835c:       05 00 00 40     sethi  %hi(0x10000), %g2
  448360:       c2 58 60 08     ldx  [ %g1 + 8 ], %g1
  448364:       80 a4 e0 00     cmp  %l3, 0
  448368:       02 c8 40 06     brz  %g1, 448380 <dma_4v_map_sg+0xc0>
  44836c:       a7 64 40 02     move  %icc, %g2, %l3
  448370:       25 00 00 08     sethi  %hi(0x2000), %l2
  448374:       a4 00 40 12     add  %g1, %l2, %l2
  448378:       a5 34 b0 0d     srlx  %l2, 0xd, %l2
  44837c:       e4 77 a7 df     stx  %l2, [ %fp + 0x7df ]
  448380:       c2 5e 21 e0     ldx  [ %i0 + 0x1e0 ], %g1
  448384:       b8 07 60 10     add  %i5, 0x10, %i4
  448388:       c2 58 40 00     ldx  [ %g1 ], %g1
  44838c:       c2 77 a7 d7     stx  %g1, [ %fp + 0x7d7 ]
  448390:       82 10 3f ff     mov  -1, %g1
  448394:       ce 5f a7 d7     ldx  [ %fp + 0x7d7 ], %g7
  448398:       83 30 70 20     srlx  %g1, 0x20, %g1
  44839c:       80 a1 c0 01     cmp  %g7, %g1
  4483a0:       b9 65 10 03     movleu  %xcc, %g3, %i4
  4483a4:       80 a6 a0 00     cmp  %i2, 0
  4483a8:       c2 5f 00 00     ldx  [ %i4 ], %g1
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  4483ac:       83 30 70 0d     srlx  %g1, 0xd, %g1
  4483b0:       04 40 01 26     ble,pn   %icc, 448848 
<dma_4v_map_sg+0x588>
  4483b4:       c2 77 a7 9f     stx  %g1, [ %fp + 0x79f ]
  4483b8:       c2 5f a7 df     ldx  [ %fp + 0x7df ], %g1
  4483bc:       84 10 3f ff     mov  -1, %g2
  4483c0:       23 00 28 21     sethi  %hi(0xa08400), %l1
  4483c4:       ce 5f a7 df     ldx  [ %fp + 0x7df ], %g7
  4483c8:       a2 14 62 b0     or  %l1, 0x2b0, %l1
  4483cc:       86 10 20 01     mov  1, %g3
  4483d0:       82 00 7f ff     add  %g1, -1, %g1
  4483d4:       e6 27 a7 af     st  %l3, [ %fp + 0x7af ]
  4483d8:       ab 30 b0 33     srlx  %g2, 0x33, %l5
  4483dc:       8e 08 40 07     and  %g1, %g7, %g7
  4483e0:       c2 77 a7 cf     stx  %g1, [ %fp + 0x7cf ]
  4483e4:       a0 10 00 19     mov  %i1, %l0
  4483e8:       f2 77 a7 a7     stx  %i1, [ %fp + 0x7a7 ]
  4483ec:       b6 10 20 00     clr  %i3
  4483f0:       a8 10 00 11     mov  %l1, %l4
  4483f4:       c6 27 a7 b3     st  %g3, [ %fp + 0x7b3 ]
  4483f8:       a6 10 00 19     mov  %i1, %l3
  4483fc:       c0 77 a7 e7     clrx  [ %fp + 0x7e7 ]
  448400:       e2 77 a7 c7     stx  %l1, [ %fp + 0x7c7 ]
  448404:       ce 77 a7 97     stx  %g7, [ %fp + 0x797 ]
  448408:       f2 77 a7 b7     stx  %i1, [ %fp + 0x7b7 ]
  44840c:       f2 04 e0 0c     ld  [ %l3 + 0xc ], %i1
  448410:       a5 36 60 00     srl  %i1, 0, %l2
  448414:       2a c4 80 1d     brnz,a,pn   %l2, 448488 
<dma_4v_map_sg+0x1c8>
  448418:       fa 5c c0 00     ldx  [ %l3 ], %i5
  44841c:       c0 77 a7 e7     clrx  [ %fp + 0x7e7 ]
  448420:       90 10 00 13     mov  %l3, %o0
  448424:       40 09 4b 47     call  69b140 <sg_next>
  448428:       b6 06 e0 01     inc  %i3
  44842c:       80 a6 c0 1a     cmp  %i3, %i2
  448430:       12 4f ff f7     bne  %icc, 44840c <dma_4v_map_sg+0x14c>
  448434:       a6 10 00 08     mov  %o0, %l3
  448438:       f2 5f a7 b7     ldx  [ %fp + 0x7b7 ], %i1
  44843c:       03 00 28 21     sethi  %hi(0xa08400), %g1
  448440:       92 10 20 a9     mov  0xa9, %o1
  448444:       82 10 62 b0     or  %g1, 0x2b0, %g1
  448448:       90 00 40 05     add  %g1, %g5, %o0
  44844c:       c2 5a 20 20     ldx  [ %o0 + 0x20 ], %g1
  448450:       80 a0 63 ff     cmp  %g1, 0x3ff
  448454:       38 60 00 5f     bgu,a,pn   %xcc, 4485d0 
<dma_4v_map_sg+0x310>
  448458:       11 00 25 01     sethi  %hi(0x940400), %o0
  44845c:       7f ff ff 29     call  448100 <iommu_batch_flush>
  448460:       d2 5f a7 d7     ldx  [ %fp + 0x7d7 ], %o1
  448464:       06 c2 00 fd     brlz,pn   %o0, 448858 
<dma_4v_map_sg+0x598>
  448468:       ce 5f a7 bf     ldx  [ %fp + 0x7bf ], %g7
  44846c:       91 91 c0 00     wrpr  %g7, %pil
  448470:       c2 07 a7 b3     ld  [ %fp + 0x7b3 ], %g1
  448474:       80 a0 40 1a     cmp  %g1, %i2
  448478:       06 40 00 e8     bl,pn   %icc, 448818 
<dma_4v_map_sg+0x558>
  44847c:       01 00 00 00     nop
  448480:       10 68 00 58     b  %xcc, 4485e0 <dma_4v_map_sg+0x320>
  448484:       f0 07 a7 b3     ld  [ %fp + 0x7b3 ], %i0
  448488:       2d 00 28 60     sethi  %hi(0xa18000), %l6
  44848c:       ae 04 80 15     add  %l2, %l5, %l7
  448490:       d4 04 e0 08     ld  [ %l3 + 8 ], %o2
  448494:       9a 10 20 00     clr  %o5
  448498:       90 10 00 18     mov  %i0, %o0
  44849c:       ba 0f 7f fc     and  %i5, -4, %i5
  4484a0:       c2 5d a2 b0     ldx  [ %l6 + 0x2b0 ], %g1
  4484a4:       92 10 00 1c     mov  %i4, %o1
  4484a8:       c4 07 60 38     ld  [ %i5 + 0x38 ], %g2
  4484ac:       96 07 a7 f7     add  %fp, 0x7f7, %o3
  4484b0:       98 10 3f ff     mov  -1, %o4
  4484b4:       fa 07 60 3c     ld  [ %i5 + 0x3c ], %i5
  4484b8:       9f 28 b0 20     sllx  %g2, 0x20, %o7
  4484bc:       84 17 40 0f     or  %i5, %o7, %g2
  4484c0:       ba 00 80 0a     add  %g2, %o2, %i5
  4484c4:       ba 27 40 01     sub  %i5, %g1, %i5
  4484c8:       84 0f 40 15     and  %i5, %l5, %g2
  4484cc:       ae 05 c0 02     add  %l7, %g2, %l7
  4484d0:       af 35 f0 0d     srlx  %l7, 0xd, %l7
  4484d4:       7f ff c5 4b     call  439a00 <iommu_tbl_range_alloc>
  4484d8:       94 10 00 17     mov  %l7, %o2
  4484dc:       80 a2 3f ff     cmp  %o0, -1
  4484e0:       02 60 00 70     be,pn   %xcc, 4486a0 
<dma_4v_map_sg+0x3e0>
  4484e4:       9a 10 00 08     mov  %o0, %o5
  4484e8:       84 04 40 05     add  %l1, %g5, %g2
  4484ec:       d8 58 a0 10     ldx  [ %g2 + 0x10 ], %o4
  4484f0:       de 58 a0 20     ldx  [ %g2 + 0x20 ], %o7
  4484f4:       9e 03 00 0f     add  %o4, %o7, %o7
  4484f8:       80 a2 00 0f     cmp  %o0, %o7
  4484fc:       02 60 00 0b     be,pn   %xcc, 448528 
<dma_4v_map_sg+0x268>
  448500:       80 a3 3f ff     cmp  %o4, -1
  448504:       02 60 00 08     be,pn   %xcc, 448524 
<dma_4v_map_sg+0x264>
  448508:       d2 5f a7 d7     ldx  [ %fp + 0x7d7 ], %o1
  44850c:       90 10 00 02     mov  %g2, %o0
  448510:       c4 77 a7 7f     stx  %g2, [ %fp + 0x77f ]
  448514:       7f ff fe fb     call  448100 <iommu_batch_flush>
  448518:       da 77 a7 67     stx  %o5, [ %fp + 0x767 ]
  44851c:       da 5f a7 67     ldx  [ %fp + 0x767 ], %o5
  448520:       c4 5f a7 7f     ldx  [ %fp + 0x77f ], %g2
  448524:       da 70 a0 10     stx  %o5, [ %g2 + 0x10 ]
  448528:       c4 5f a7 c7     ldx  [ %fp + 0x7c7 ], %g2
  44852c:       86 10 3f ff     mov  -1, %g3
  448530:       ae 05 ff ff     add  %l7, -1, %l7
  448534:       9f 28 f0 0d     sllx  %g3, 0xd, %o7
  448538:       d8 5f 00 00     ldx  [ %i4 ], %o4
  44853c:       90 00 80 05     add  %g2, %g5, %o0
  448540:       ba 0f 40 0f     and  %i5, %o7, %i5
  448544:       c4 5a 20 20     ldx  [ %o0 + 0x20 ], %g2
  448548:       80 a0 a3 ff     cmp  %g2, 0x3ff
  44854c:       18 60 00 1f     bgu,pn   %xcc, 4485c8 
<dma_4v_map_sg+0x308>
  448550:       d4 04 e0 08     ld  [ %l3 + 8 ], %o2
  448554:       82 10 00 11     mov  %l1, %g1
  448558:       f2 27 a7 93     st  %i1, [ %fp + 0x793 ]
  44855c:       a2 10 00 1b     mov  %i3, %l1
  448560:       b2 10 00 1a     mov  %i2, %i1
  448564:       b6 10 00 1c     mov  %i4, %i3
  448568:       b4 10 00 14     mov  %l4, %i2
  44856c:       b8 10 00 17     mov  %l7, %i4
  448570:       a8 10 00 18     mov  %i0, %l4
  448574:       17 00 00 08     sethi  %hi(0x2000), %o3
  448578:       b0 10 00 0c     mov  %o4, %i0
  44857c:       ae 10 00 01     mov  %g1, %l7
  448580:       c2 5a 20 18     ldx  [ %o0 + 0x18 ], %g1
  448584:       9e 00 a0 01     add  %g2, 1, %o7
  448588:       85 28 b0 03     sllx  %g2, 3, %g2
  44858c:       de 72 20 20     stx  %o7, [ %o0 + 0x20 ]
  448590:       fa 70 40 02     stx  %i5, [ %g1 + %g2 ]
  448594:       c2 5a 20 20     ldx  [ %o0 + 0x20 ], %g1
  448598:       80 a0 64 00     cmp  %g1, 0x400
  44859c:       02 60 00 13     be,pn   %xcc, 4485e8 
<dma_4v_map_sg+0x328>
  4485a0:       d2 5f a7 d7     ldx  [ %fp + 0x7d7 ], %o1
  4485a4:       b8 07 3f ff     add  %i4, -1, %i4
  4485a8:       80 a7 3f ff     cmp  %i4, -1
  4485ac:       02 60 00 4f     be,pn   %xcc, 4486e8 
<dma_4v_map_sg+0x428>
  4485b0:       ba 07 40 0b     add  %i5, %o3, %i5
  4485b4:       90 06 80 05     add  %i2, %g5, %o0
  4485b8:       c4 5a 20 20     ldx  [ %o0 + 0x20 ], %g2
  4485bc:       80 a0 a3 ff     cmp  %g2, 0x3ff
  4485c0:       28 6f ff f1     bleu,a   %xcc, 448584 
<dma_4v_map_sg+0x2c4>
  4485c4:       c2 5a 20 18     ldx  [ %o0 + 0x18 ], %g1
  4485e0:       81 cf e0 08     rett  %i7 + 8
  4485e4:       91 3a 20 00     sra  %o0, 0, %o0
  4485e8:       d4 77 a7 77     stx  %o2, [ %fp + 0x777 ]
  4485ec:       d6 77 a7 6f     stx  %o3, [ %fp + 0x76f ]
  4485f0:       7f ff fe c4     call  448100 <iommu_batch_flush>
  4485f4:       da 77 a7 67     stx  %o5, [ %fp + 0x767 ]
  4485f8:       d4 5f a7 77     ldx  [ %fp + 0x777 ], %o2
  4485fc:       d6 5f a7 6f     ldx  [ %fp + 0x76f ], %o3
  448600:       0e f2 3f e9     brgez,pn   %o0, 4485a4 
<dma_4v_map_sg+0x2e4>
  448604:       da 5f a7 67     ldx  [ %fp + 0x767 ], %o5
  448608:       b4 10 00 19     mov  %i1, %i2
  44860c:       f2 5f a7 b7     ldx  [ %fp + 0x7b7 ], %i1
  448610:       b8 10 00 1b     mov  %i3, %i4
  448614:       b6 10 3f ff     mov  -1, %i3
  448618:       ba 10 20 00     clr  %i5
  44861c:       b1 2e f0 0d     sllx  %i3, 0xd, %i0
  448620:       b7 36 f0 33     srlx  %i3, 0x33, %i3
  448624:       d4 06 60 18     ld  [ %i1 + 0x18 ], %o2
  448628:       80 a2 a0 00     cmp  %o2, 0
  44862c:       32 40 00 10     bne,a,pn   %icc, 44866c 
<dma_4v_map_sg+0x3ac>
  448630:       d2 5e 60 10     ldx  [ %i1 + 0x10 ], %o1
  448634:       80 a6 40 10     cmp  %i1, %l0
  448638:       02 60 00 09     be,pn   %xcc, 44865c 
<dma_4v_map_sg+0x39c>
  44863c:       c2 5f a7 bf     ldx  [ %fp + 0x7bf ], %g1
  448640:       90 10 00 19     mov  %i1, %o0
  448644:       40 09 4a bf     call  69b140 <sg_next>
  448648:       ba 07 60 01     inc  %i5
  44864c:       80 a7 40 1a     cmp  %i5, %i2
  448650:       06 4f ff f5     bl  %icc, 448624 <dma_4v_map_sg+0x364>
  448654:       b2 10 00 08     mov  %o0, %i1
  448658:       c2 5f a7 bf     ldx  [ %fp + 0x7bf ], %g1
  44865c:       91 90 40 00     wrpr  %g1, %pil
  448660:       b0 10 20 00     clr  %i0
  448664:       81 cf e0 08     rett  %i7 + 8
  448668:       91 3a 20 00     sra  %o0, 0, %o0
  44866c:       95 32 a0 00     srl  %o2, 0, %o2
  448670:       90 10 00 1c     mov  %i4, %o0
  448674:       96 10 3f ff     mov  -1, %o3
  448678:       82 0a 40 1b     and  %o1, %i3, %g1
  44867c:       92 0a 40 18     and  %o1, %i0, %o1
  448680:       94 02 80 01     add  %o2, %g1, %o2
  448684:       94 02 80 1b     add  %o2, %i3, %o2
  448688:       7f ff c5 c6     call  439da0 <iommu_tbl_range_free>
  44868c:       95 32 b0 0d     srlx  %o2, 0xd, %o2
  448690:       86 10 3f ff     mov  -1, %g3
  448694:       c0 26 60 18     clr  [ %i1 + 0x18 ]
  448698:       10 6f ff e7     b  %xcc, 448634 <dma_4v_map_sg+0x374>
  44869c:       c6 76 60 10     stx  %g3, [ %i1 + 0x10 ]
  4486a0:       11 00 26 b3     sethi  %hi(0x9acc00), %o0
  4486a4:       13 00 22 e3     sethi  %hi(0x8b8c00), %o1
.....



_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu