Re: [PATCH/RFC 0/4] dma ops and virtio

[Cornelia Huck]

On Thu, 5 Nov 2015 09:11:06 +0100
Cornelia Huck  wrote:

> On Wed, 4 Nov 2015 10:14:11 -0800
> Andy Lutomirski  wrote:
> 
> > On Wed, Nov 4, 2015 at 9:52 AM, Cornelia Huck  
> > wrote:
> > > On Wed, 4 Nov 2015 15:29:23 +0100
> > > Cornelia Huck  wrote:
> > >
> > >> I'm currently suspecting some endianness issues, probably with the ecw
> > >> accesses, which is why I'd be interested in your trace information (as
> > >> I currently don't have a LE test setup at hand.)
> > >
> > > I think I've got it. We have sense_data as a byte array, which
> > > implicitly makes it BE already. When we copy to the ecws while building
> > > the irb, the data ends up in 32 bit values. The conversion from host
> > > endianness to BE now treats them as LE on your system...
> > >
> > > Could you please give the following qemu patch a try?
> > 
> > Tested-by: Andy Lutomirski 
> > 
> > Now my test script panics for the right reason (init isn't actually an
> > s390 binary).  Thanks!
> 
> Cool, thanks for testing! I'll get this into qemu as a proper patch.

FYI: The patch has been included into the qemu git tree.

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH/RFC 0/4] dma ops and virtio

[Cornelia Huck]

On Wed, 4 Nov 2015 10:14:11 -0800
Andy Lutomirski  wrote:

> On Wed, Nov 4, 2015 at 9:52 AM, Cornelia Huck  
> wrote:
> > On Wed, 4 Nov 2015 15:29:23 +0100
> > Cornelia Huck  wrote:
> >
> >> I'm currently suspecting some endianness issues, probably with the ecw
> >> accesses, which is why I'd be interested in your trace information (as
> >> I currently don't have a LE test setup at hand.)
> >
> > I think I've got it. We have sense_data as a byte array, which
> > implicitly makes it BE already. When we copy to the ecws while building
> > the irb, the data ends up in 32 bit values. The conversion from host
> > endianness to BE now treats them as LE on your system...
> >
> > Could you please give the following qemu patch a try?
> 
> Tested-by: Andy Lutomirski 
> 
> Now my test script panics for the right reason (init isn't actually an
> s390 binary).  Thanks!

Cool, thanks for testing! I'll get this into qemu as a proper patch.

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH/RFC 0/4] dma ops and virtio

[Cornelia Huck]

On Tue, 3 Nov 2015 10:45:12 -0800
Andy Lutomirski  wrote:

> On Tue, Nov 3, 2015 at 9:59 AM, Cornelia Huck  
> wrote:
> > It's just failing very early in the setup phase. As it works for me
> > with a kvm setup, I'm suspecting some error in qemu's emulation code,
> > which is unfortunately not my turf.
> >
> 
> That should be easy to rule out.  Can you try with -machine accel=tcg?
>  I can't test with kvm for obvious reasons.

Well, s390-on-s390 works (at least with
https://patchwork.ozlabs.org/patch/538882/ applied).

I'm currently suspecting some endianness issues, probably with the ecw
accesses, which is why I'd be interested in your trace information (as
I currently don't have a LE test setup at hand.)

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH/RFC 0/4] dma ops and virtio

[Cornelia Huck]

On Wed, 4 Nov 2015 15:29:23 +0100
Cornelia Huck  wrote:

> I'm currently suspecting some endianness issues, probably with the ecw
> accesses, which is why I'd be interested in your trace information (as
> I currently don't have a LE test setup at hand.)

I think I've got it. We have sense_data as a byte array, which
implicitly makes it BE already. When we copy to the ecws while building
the irb, the data ends up in 32 bit values. The conversion from host
endianness to BE now treats them as LE on your system...

Could you please give the following qemu patch a try?

diff --git a/hw/s390x/css.c b/hw/s390x/css.c
index c033612..a13494e 100644
--- a/hw/s390x/css.c
+++ b/hw/s390x/css.c
@@ -892,8 +892,16 @@ int css_do_tsch_get_irb(SubchDev *sch, IRB *target_irb, 
int *irb_len)
 /* If a unit check is pending, copy sense data. */
 if ((s->dstat & SCSW_DSTAT_UNIT_CHECK) &&
 (p->chars & PMCW_CHARS_MASK_CSENSE)) {
+uint32_t ecw[8];
+int i;
+
 irb.scsw.flags |= SCSW_FLAGS_MASK_ESWF | SCSW_FLAGS_MASK_ECTL;
-memcpy(irb.ecw, sch->sense_data, sizeof(sch->sense_data));
+/* Attention: sense_data is already BE! */
+memset(ecw, 0, sizeof(ecw));
+memcpy(ecw, sch->sense_data, sizeof(sch->sense_data));
+for (i = 0; i < ARRAY_SIZE(ecw); i++) {
+irb.ecw[i] = be32_to_cpu(ecw[i]);
+}
 irb.esw[1] = 0x0100 | (sizeof(sch->sense_data) << 8);
 }
 }

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH/RFC 0/4] dma ops and virtio

[Andy Lutomirski]

On Wed, Nov 4, 2015 at 9:52 AM, Cornelia Huck  wrote:
> On Wed, 4 Nov 2015 15:29:23 +0100
> Cornelia Huck  wrote:
>
>> I'm currently suspecting some endianness issues, probably with the ecw
>> accesses, which is why I'd be interested in your trace information (as
>> I currently don't have a LE test setup at hand.)
>
> I think I've got it. We have sense_data as a byte array, which
> implicitly makes it BE already. When we copy to the ecws while building
> the irb, the data ends up in 32 bit values. The conversion from host
> endianness to BE now treats them as LE on your system...
>
> Could you please give the following qemu patch a try?

Tested-by: Andy Lutomirski 

Now my test script panics for the right reason (init isn't actually an
s390 binary).  Thanks!

I'll update my branch with the latest s390 DMA stuff, and now I can
give it at least basic regression testing.  Of course, I have to find
a root filesystem, too. :)

--Andy
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH/RFC 0/4] dma ops and virtio

[Christian Borntraeger]

Am 02.11.2015 um 21:23 schrieb Andy Lutomirski:
> On Mon, Nov 2, 2015 at 3:16 AM, Cornelia Huck  
> wrote:
>> On Fri, 30 Oct 2015 13:33:07 -0700
>> Andy Lutomirski  wrote:
>>
>>> On Fri, Oct 30, 2015 at 1:25 AM, Cornelia Huck  
>>> wrote:
 On Thu, 29 Oct 2015 15:50:38 -0700
 Andy Lutomirski  wrote:

> Progress!  After getting that sort-of-working, I figured out what was
> wrong with my earlier command, and I got that working, too.  Now I
> get:
>
> qemu-system-s390x -fsdev
> local,id=virtfs1,path=/,security_model=none,readonly -device
> virtio-9p-ccw,fsdev=virtfs1,mount_tag=/dev/root -M s390-ccw-virtio
> -nodefaults -device sclpconsole,chardev=console -parallel none -net
> none -echr 1 -serial none -chardev stdio,id=console,signal=off,mux=on
> -serial chardev:console -mon chardev=console -vga none -display none
> -kernel arch/s390/boot/bzImage -append
> 'init=/home/luto/devel/virtme/virtme/guest/virtme-init
> psmouse.proto=exps "virtme_stty_con=rows 24 cols 150 iutf8"
> TERM=xterm-256color rootfstype=9p
> rootflags=ro,version=9p2000.L,trans=virtio,access=any
> raid=noautodetect debug'

 The commandline looks sane AFAICS.

 (...)

> vrfy: device 0.0.: rc=0 pgroup=0 mpath=0 vpm=80
> virtio_ccw 0.0.: Failed to set online: -5
>
> ^^^ bad news!

 I'd like to see where in the onlining process this fails. Could you set
 up qemu tracing for css_* and virtio_ccw_* (instructions in
 qemu/docs/tracing.txt)?
>>>
>>> I have a file called events that contains:
>>>
>>> css_*
>>> virtio_ccw_*
>>>
>>> pointing -trace events= at it results in a trace- file that's 549
>>> bytes long and contains nothing.  Are wildcards not as well-supported
>>> as the docs suggest?
>>
>> Just tried it, seemed to work for me as expected. And as your messages
>> indicate, at least some of the css tracepoints are guaranteed to be
>> hit. Odd.
>>
>> Can you try the following sophisticated printf debug patch?
>>
>> diff --git a/hw/s390x/css.c b/hw/s390x/css.c
>> index c033612..6a87bd6 100644
>> --- a/hw/s390x/css.c
>> +++ b/hw/s390x/css.c
>> @@ -308,6 +308,8 @@ static int css_interpret_ccw(SubchDev *sch, hwaddr 
>> ccw_addr)
>>  sch->ccw_no_data_cnt++;
>>  }
>>
>> +fprintf(stderr, "CH DBG: %s: cmd_code=%x\n", __func__, ccw.cmd_code);
>> +
>>  /* Look at the command. */
>>  switch (ccw.cmd_code) {
>>  case CCW_CMD_NOOP:
>> @@ -375,6 +377,7 @@ static int css_interpret_ccw(SubchDev *sch, hwaddr 
>> ccw_addr)
>>  }
>>  break;
>>  }
>> +fprintf(stderr, "CH DBG: %s: ret=%d\n", __func__, ret);
>>  sch->last_cmd = ccw;
>>  sch->last_cmd_valid = true;
>>  if (ret == 0) {
>>
>>
 Which qemu version is this, btw.?

>>>
>>> git from yesterday.
>>
>> Hm. Might be worth trying the s390-ccw-virtio-2.4 machine instead.
>>
> 
> No change.
> 
> With s390-ccw-virtio-2.4, I get:
> 
> Initializing cgroup subsys cpuset
> Initializing cgroup subsys cpu
> Initializing cgroup subsys cpuacct
> Linux version 4.3.0-rc7-8-gff230d6ec6b2
> (l...@amaluto.corp.amacapital.net) (gcc version 5.1.1 20150618 (Red
> Hat Cross 5.1.1-3) (GCC) ) #344 SMP Fri Oct 30 13:16:13 PDT 2015
> setup: Linux is running under KVM in 64-bit mode
> setup: Max memory size: 128MB
> Zone ranges:
>   DMA  [mem 0x-0x7fff]
>   Normal   empty
> Movable zone start for each node
> Early memory node ranges
>   node   0: [mem 0x-0x07ff]
> Initmem setup node 0 [mem 0x-0x07ff]
> On node 0 totalpages: 32768
>   DMA zone: 512 pages used for memmap
>   DMA zone: 0 pages reserved
>   DMA zone: 32768 pages, LIFO batch:7
> PERCPU: Embedded 466 pages/cpu @07605000 s1868032 r8192 d32512 
> u1908736
> pcpu-alloc: s1868032 r8192 d32512 u1908736 alloc=466*4096
> pcpu-alloc: [0] 0 [0] 1
> Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 32256
> Kernel command line:
> init=/home/luto/devel/virtme/virtme/guest/virtme-init
> psmouse.proto=exps "virtme_stty_con=rows 45 cols 150 iutf8"
> TERM=xterm-256color rootfstype=9p
> rootflags=version=9p2000.L,trans=virtio,access=any raid=noautodetect
> ro debug
> PID hash table entries: 512 (order: 0, 4096 bytes)
> Dentry cache hash table entries: 16384 (order: 5, 131072 bytes)
> Inode-cache hash table entries: 8192 (order: 4, 65536 bytes)
> Memory: 92520K/131072K available (8255K kernel code, 802K rwdata,


can you send your kernel config?

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH/RFC 0/4] dma ops and virtio

[Andy Lutomirski]

On Tue, Nov 3, 2015 at 12:14 AM, Christian Borntraeger
 wrote:
> Am 02.11.2015 um 21:23 schrieb Andy Lutomirski:
>> On Mon, Nov 2, 2015 at 3:16 AM, Cornelia Huck  
>> wrote:
>>> On Fri, 30 Oct 2015 13:33:07 -0700
>>> Andy Lutomirski  wrote:
>>>
 On Fri, Oct 30, 2015 at 1:25 AM, Cornelia Huck  
 wrote:
> On Thu, 29 Oct 2015 15:50:38 -0700
> Andy Lutomirski  wrote:
>
>> Progress!  After getting that sort-of-working, I figured out what was
>> wrong with my earlier command, and I got that working, too.  Now I
>> get:
>>
>> qemu-system-s390x -fsdev
>> local,id=virtfs1,path=/,security_model=none,readonly -device
>> virtio-9p-ccw,fsdev=virtfs1,mount_tag=/dev/root -M s390-ccw-virtio
>> -nodefaults -device sclpconsole,chardev=console -parallel none -net
>> none -echr 1 -serial none -chardev stdio,id=console,signal=off,mux=on
>> -serial chardev:console -mon chardev=console -vga none -display none
>> -kernel arch/s390/boot/bzImage -append
>> 'init=/home/luto/devel/virtme/virtme/guest/virtme-init
>> psmouse.proto=exps "virtme_stty_con=rows 24 cols 150 iutf8"
>> TERM=xterm-256color rootfstype=9p
>> rootflags=ro,version=9p2000.L,trans=virtio,access=any
>> raid=noautodetect debug'
>
> The commandline looks sane AFAICS.
>
> (...)
>
>> vrfy: device 0.0.: rc=0 pgroup=0 mpath=0 vpm=80
>> virtio_ccw 0.0.: Failed to set online: -5
>>
>> ^^^ bad news!
>
> I'd like to see where in the onlining process this fails. Could you set
> up qemu tracing for css_* and virtio_ccw_* (instructions in
> qemu/docs/tracing.txt)?

 I have a file called events that contains:

 css_*
 virtio_ccw_*

 pointing -trace events= at it results in a trace- file that's 549
 bytes long and contains nothing.  Are wildcards not as well-supported
 as the docs suggest?
>>>
>>> Just tried it, seemed to work for me as expected. And as your messages
>>> indicate, at least some of the css tracepoints are guaranteed to be
>>> hit. Odd.
>>>
>>> Can you try the following sophisticated printf debug patch?
>>>
>>> diff --git a/hw/s390x/css.c b/hw/s390x/css.c
>>> index c033612..6a87bd6 100644
>>> --- a/hw/s390x/css.c
>>> +++ b/hw/s390x/css.c
>>> @@ -308,6 +308,8 @@ static int css_interpret_ccw(SubchDev *sch, hwaddr 
>>> ccw_addr)
>>>  sch->ccw_no_data_cnt++;
>>>  }
>>>
>>> +fprintf(stderr, "CH DBG: %s: cmd_code=%x\n", __func__, ccw.cmd_code);
>>> +
>>>  /* Look at the command. */
>>>  switch (ccw.cmd_code) {
>>>  case CCW_CMD_NOOP:
>>> @@ -375,6 +377,7 @@ static int css_interpret_ccw(SubchDev *sch, hwaddr 
>>> ccw_addr)
>>>  }
>>>  break;
>>>  }
>>> +fprintf(stderr, "CH DBG: %s: ret=%d\n", __func__, ret);
>>>  sch->last_cmd = ccw;
>>>  sch->last_cmd_valid = true;
>>>  if (ret == 0) {
>>>
>>>
> Which qemu version is this, btw.?
>

 git from yesterday.
>>>
>>> Hm. Might be worth trying the s390-ccw-virtio-2.4 machine instead.
>>>
>>
>> No change.
>>
>> With s390-ccw-virtio-2.4, I get:
>>
>> Initializing cgroup subsys cpuset
>> Initializing cgroup subsys cpu
>> Initializing cgroup subsys cpuacct
>> Linux version 4.3.0-rc7-8-gff230d6ec6b2
>> (l...@amaluto.corp.amacapital.net) (gcc version 5.1.1 20150618 (Red
>> Hat Cross 5.1.1-3) (GCC) ) #344 SMP Fri Oct 30 13:16:13 PDT 2015
>> setup: Linux is running under KVM in 64-bit mode
>> setup: Max memory size: 128MB
>> Zone ranges:
>>   DMA  [mem 0x-0x7fff]
>>   Normal   empty
>> Movable zone start for each node
>> Early memory node ranges
>>   node   0: [mem 0x-0x07ff]
>> Initmem setup node 0 [mem 0x-0x07ff]
>> On node 0 totalpages: 32768
>>   DMA zone: 512 pages used for memmap
>>   DMA zone: 0 pages reserved
>>   DMA zone: 32768 pages, LIFO batch:7
>> PERCPU: Embedded 466 pages/cpu @07605000 s1868032 r8192 d32512 
>> u1908736
>> pcpu-alloc: s1868032 r8192 d32512 u1908736 alloc=466*4096
>> pcpu-alloc: [0] 0 [0] 1
>> Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 32256
>> Kernel command line:
>> init=/home/luto/devel/virtme/virtme/guest/virtme-init
>> psmouse.proto=exps "virtme_stty_con=rows 45 cols 150 iutf8"
>> TERM=xterm-256color rootfstype=9p
>> rootflags=version=9p2000.L,trans=virtio,access=any raid=noautodetect
>> ro debug
>> PID hash table entries: 512 (order: 0, 4096 bytes)
>> Dentry cache hash table entries: 16384 (order: 5, 131072 bytes)
>> Inode-cache hash table entries: 8192 (order: 4, 65536 bytes)
>> Memory: 92520K/131072K available (8255K kernel code, 802K rwdata,
>
>
> can you send your kernel config?
>

Attached.

A failing command looks like:

qemu-system-s390x -fsdev
local,id=virtfs1,path=/,security_model=none,readonly 

Re: [PATCH/RFC 0/4] dma ops and virtio

[Cornelia Huck]

On Mon, 2 Nov 2015 12:23:25 -0800
Andy Lutomirski  wrote:

> No change.

I'm stumped :(

Here's what I see:

(...)

> CH DBG: css_interpret_ccw: cmd_code=e4
> CH DBG: css_interpret_ccw: ret=0

sense id -> works

(...)

> CH DBG: css_interpret_ccw: cmd_code=3
> CH DBG: css_interpret_ccw: ret=0

nop (path verification) -> works

> CH DBG: css_interpret_ccw: cmd_code=83
> CH DBG: css_interpret_ccw: ret=-38

set revision -> -ENOSYS

This is fine; the virtio device is in legacy mode and the kernel will
try revision 1; qemu will reject this. The code should end up
generating a unit check with command reject, however...

> qeth: register layer 2 discipline
> qeth: register layer 3 discipline
> oprofile: using timer interrupt.
> NET: Registered protocol family 10
> virtio_ccw 0.0.: Failed to set online: -5

...this shows the kernel driver somehow did not end up with that
command reject (it would have triggered a retry with revision 0, and
the return code shows that no unit check/command reject was detected,
but some other error.)

> The lack of much interesting output makes me think that maybe I
> misconfigured something.

It's just failing very early in the setup phase. As it works for me
with a kvm setup, I'm suspecting some error in qemu's emulation code,
which is unfortunately not my turf.

Some more poke-around-in-the-dark ideas:

- Do you get more debug out put when you switch back to s390-ccw-virtio
(virtio-1), i.e. does cmd 83 work and is it followed by further
commands?

- Can you try with the following qemu logging patch

-8<--8<-

diff --git a/hw/s390x/css.c b/hw/s390x/css.c
index c033612..80853a6 100644
--- a/hw/s390x/css.c
+++ b/hw/s390x/css.c
@@ -868,6 +868,7 @@ int css_do_tsch_get_irb(SubchDev *sch, IRB *target_irb, int 
*irb_len)
 PMCW *p = >curr_status.pmcw;
 uint16_t stctl;
 IRB irb;
+int i;
 
 if (!(p->flags & (PMCW_FLAGS_MASK_DNV | PMCW_FLAGS_MASK_ENA))) {
 return 3;
@@ -898,6 +899,14 @@ int css_do_tsch_get_irb(SubchDev *sch, IRB *target_irb, 
int *irb_len)
 }
 }
 /* Store the irb to the guest. */
+fprintf(stderr, "CH DBG: %s: flags=%04x ctrl=%04x cpa=%08x\n",
+__func__, irb.scsw.flags, irb.scsw.ctrl, irb.scsw.cpa);
+fprintf(stderr, "CH DBG: %s: dstat=%02x cstat=%02x count=%04x\n",
+__func__, irb.scsw.dstat, irb.scsw.cstat, irb.scsw.count);
+for (i = 0; i < ARRAY_SIZE(irb.ecw); i++) {
+fprintf(stderr, "CH DBG: %s: ecw[%d]=%08x\n", __func__,
+i, irb.ecw[i]);
+}
 copy_irb_to_guest(target_irb, , p, irb_len);
 
 return ((stctl & SCSW_STCTL_STATUS_PEND) == 0);

-8<--8<-

and the following kernel patch

-8<--8<-

diff --git a/drivers/s390/cio/device_fsm.c b/drivers/s390/cio/device_fsm.c
index 83da53c..ea4db09 100644
--- a/drivers/s390/cio/device_fsm.c
+++ b/drivers/s390/cio/device_fsm.c
@@ -540,6 +540,9 @@ callback:
create_fake_irb(>private->irb,
cdev->private->flags.fake_irb);
cdev->private->flags.fake_irb = 0;
+   CIO_TRACE_EVENT(0, "fake_irb");
+   CIO_HEX_EVENT(0, >private->irb,
+ sizeof(struct irb));
if (cdev->handler)
cdev->handler(cdev, cdev->private->intparm,
  >private->irb);
diff --git a/drivers/s390/cio/device_ops.c b/drivers/s390/cio/device_ops.c
index 6acd0b5..e9bf357 100644
--- a/drivers/s390/cio/device_ops.c
+++ b/drivers/s390/cio/device_ops.c
@@ -446,6 +446,8 @@ ccw_device_call_handler(struct ccw_device *cdev)
/*
 * Now we are ready to call the device driver interrupt handler.
 */
+   CIO_TRACE_EVENT(0, "irb");
+   CIO_HEX_EVENT(0, >private->irb, sizeof(struct irb));
if (cdev->handler)
cdev->handler(cdev, cdev->private->intparm,
  >private->irb);

-8<--8<-

Just to verify that qemu will produce and the kernel end up with the
irb I'd expect. I'd rather prefer us getting the dma stuff right
instead of chasing qemu issues :/

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH/RFC 0/4] dma ops and virtio

[Andy Lutomirski]

On Tue, Nov 3, 2015 at 9:59 AM, Cornelia Huck  wrote:
> It's just failing very early in the setup phase. As it works for me
> with a kvm setup, I'm suspecting some error in qemu's emulation code,
> which is unfortunately not my turf.
>

That should be easy to rule out.  Can you try with -machine accel=tcg?
 I can't test with kvm for obvious reasons.

> Some more poke-around-in-the-dark ideas:
>
> - Do you get more debug out put when you switch back to s390-ccw-virtio
> (virtio-1), i.e. does cmd 83 work and is it followed by further
> commands?

I'll play with this stuff later today.

--Andy
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH/RFC 0/4] dma ops and virtio

[Andy Lutomirski]

On Mon, Nov 2, 2015 at 3:16 AM, Cornelia Huck  wrote:
> On Fri, 30 Oct 2015 13:33:07 -0700
> Andy Lutomirski  wrote:
>
>> On Fri, Oct 30, 2015 at 1:25 AM, Cornelia Huck  
>> wrote:
>> > On Thu, 29 Oct 2015 15:50:38 -0700
>> > Andy Lutomirski  wrote:
>> >
>> >> Progress!  After getting that sort-of-working, I figured out what was
>> >> wrong with my earlier command, and I got that working, too.  Now I
>> >> get:
>> >>
>> >> qemu-system-s390x -fsdev
>> >> local,id=virtfs1,path=/,security_model=none,readonly -device
>> >> virtio-9p-ccw,fsdev=virtfs1,mount_tag=/dev/root -M s390-ccw-virtio
>> >> -nodefaults -device sclpconsole,chardev=console -parallel none -net
>> >> none -echr 1 -serial none -chardev stdio,id=console,signal=off,mux=on
>> >> -serial chardev:console -mon chardev=console -vga none -display none
>> >> -kernel arch/s390/boot/bzImage -append
>> >> 'init=/home/luto/devel/virtme/virtme/guest/virtme-init
>> >> psmouse.proto=exps "virtme_stty_con=rows 24 cols 150 iutf8"
>> >> TERM=xterm-256color rootfstype=9p
>> >> rootflags=ro,version=9p2000.L,trans=virtio,access=any
>> >> raid=noautodetect debug'
>> >
>> > The commandline looks sane AFAICS.
>> >
>> > (...)
>> >
>> >> vrfy: device 0.0.: rc=0 pgroup=0 mpath=0 vpm=80
>> >> virtio_ccw 0.0.: Failed to set online: -5
>> >>
>> >> ^^^ bad news!
>> >
>> > I'd like to see where in the onlining process this fails. Could you set
>> > up qemu tracing for css_* and virtio_ccw_* (instructions in
>> > qemu/docs/tracing.txt)?
>>
>> I have a file called events that contains:
>>
>> css_*
>> virtio_ccw_*
>>
>> pointing -trace events= at it results in a trace- file that's 549
>> bytes long and contains nothing.  Are wildcards not as well-supported
>> as the docs suggest?
>
> Just tried it, seemed to work for me as expected. And as your messages
> indicate, at least some of the css tracepoints are guaranteed to be
> hit. Odd.
>
> Can you try the following sophisticated printf debug patch?
>
> diff --git a/hw/s390x/css.c b/hw/s390x/css.c
> index c033612..6a87bd6 100644
> --- a/hw/s390x/css.c
> +++ b/hw/s390x/css.c
> @@ -308,6 +308,8 @@ static int css_interpret_ccw(SubchDev *sch, hwaddr 
> ccw_addr)
>  sch->ccw_no_data_cnt++;
>  }
>
> +fprintf(stderr, "CH DBG: %s: cmd_code=%x\n", __func__, ccw.cmd_code);
> +
>  /* Look at the command. */
>  switch (ccw.cmd_code) {
>  case CCW_CMD_NOOP:
> @@ -375,6 +377,7 @@ static int css_interpret_ccw(SubchDev *sch, hwaddr 
> ccw_addr)
>  }
>  break;
>  }
> +fprintf(stderr, "CH DBG: %s: ret=%d\n", __func__, ret);
>  sch->last_cmd = ccw;
>  sch->last_cmd_valid = true;
>  if (ret == 0) {
>
>
>> > Which qemu version is this, btw.?
>> >
>>
>> git from yesterday.
>
> Hm. Might be worth trying the s390-ccw-virtio-2.4 machine instead.
>

No change.

With s390-ccw-virtio-2.4, I get:

Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Initializing cgroup subsys cpuacct
Linux version 4.3.0-rc7-8-gff230d6ec6b2
(l...@amaluto.corp.amacapital.net) (gcc version 5.1.1 20150618 (Red
Hat Cross 5.1.1-3) (GCC) ) #344 SMP Fri Oct 30 13:16:13 PDT 2015
setup: Linux is running under KVM in 64-bit mode
setup: Max memory size: 128MB
Zone ranges:
  DMA  [mem 0x-0x7fff]
  Normal   empty
Movable zone start for each node
Early memory node ranges
  node   0: [mem 0x-0x07ff]
Initmem setup node 0 [mem 0x-0x07ff]
On node 0 totalpages: 32768
  DMA zone: 512 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 32768 pages, LIFO batch:7
PERCPU: Embedded 466 pages/cpu @07605000 s1868032 r8192 d32512 u1908736
pcpu-alloc: s1868032 r8192 d32512 u1908736 alloc=466*4096
pcpu-alloc: [0] 0 [0] 1
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 32256
Kernel command line:
init=/home/luto/devel/virtme/virtme/guest/virtme-init
psmouse.proto=exps "virtme_stty_con=rows 45 cols 150 iutf8"
TERM=xterm-256color rootfstype=9p
rootflags=version=9p2000.L,trans=virtio,access=any raid=noautodetect
ro debug
PID hash table entries: 512 (order: 0, 4096 bytes)
Dentry cache hash table entries: 16384 (order: 5, 131072 bytes)
Inode-cache hash table entries: 8192 (order: 4, 65536 bytes)
Memory: 92520K/131072K available (8255K kernel code, 802K rwdata,
3860K rodata, 2384K init, 14382K bss, 38552K reserved, 0K
cma-reserved)
Write protected kernel read-only data: 0x10 - 0xcd4fff
SLUB: HWalign=256, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
Running RCU self tests
Hierarchical RCU implementation.
RCU debugfs-based tracing is enabled.
RCU lockdep checking is enabled.
Build-time adjustment of leaf fanout to 64.
RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=2.
RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=2
NR_IRQS:3
clocksource: tod: mask: 0x max_cycles:

Re: [PATCH/RFC 0/4] dma ops and virtio

[Cornelia Huck]

On Fri, 30 Oct 2015 13:33:07 -0700
Andy Lutomirski  wrote:

> On Fri, Oct 30, 2015 at 1:25 AM, Cornelia Huck  
> wrote:
> > On Thu, 29 Oct 2015 15:50:38 -0700
> > Andy Lutomirski  wrote:
> >
> >> Progress!  After getting that sort-of-working, I figured out what was
> >> wrong with my earlier command, and I got that working, too.  Now I
> >> get:
> >>
> >> qemu-system-s390x -fsdev
> >> local,id=virtfs1,path=/,security_model=none,readonly -device
> >> virtio-9p-ccw,fsdev=virtfs1,mount_tag=/dev/root -M s390-ccw-virtio
> >> -nodefaults -device sclpconsole,chardev=console -parallel none -net
> >> none -echr 1 -serial none -chardev stdio,id=console,signal=off,mux=on
> >> -serial chardev:console -mon chardev=console -vga none -display none
> >> -kernel arch/s390/boot/bzImage -append
> >> 'init=/home/luto/devel/virtme/virtme/guest/virtme-init
> >> psmouse.proto=exps "virtme_stty_con=rows 24 cols 150 iutf8"
> >> TERM=xterm-256color rootfstype=9p
> >> rootflags=ro,version=9p2000.L,trans=virtio,access=any
> >> raid=noautodetect debug'
> >
> > The commandline looks sane AFAICS.
> >
> > (...)
> >
> >> vrfy: device 0.0.: rc=0 pgroup=0 mpath=0 vpm=80
> >> virtio_ccw 0.0.: Failed to set online: -5
> >>
> >> ^^^ bad news!
> >
> > I'd like to see where in the onlining process this fails. Could you set
> > up qemu tracing for css_* and virtio_ccw_* (instructions in
> > qemu/docs/tracing.txt)?
> 
> I have a file called events that contains:
> 
> css_*
> virtio_ccw_*
> 
> pointing -trace events= at it results in a trace- file that's 549
> bytes long and contains nothing.  Are wildcards not as well-supported
> as the docs suggest?

Just tried it, seemed to work for me as expected. And as your messages
indicate, at least some of the css tracepoints are guaranteed to be
hit. Odd.

Can you try the following sophisticated printf debug patch?

diff --git a/hw/s390x/css.c b/hw/s390x/css.c
index c033612..6a87bd6 100644
--- a/hw/s390x/css.c
+++ b/hw/s390x/css.c
@@ -308,6 +308,8 @@ static int css_interpret_ccw(SubchDev *sch, hwaddr ccw_addr)
 sch->ccw_no_data_cnt++;
 }
 
+fprintf(stderr, "CH DBG: %s: cmd_code=%x\n", __func__, ccw.cmd_code);
+
 /* Look at the command. */
 switch (ccw.cmd_code) {
 case CCW_CMD_NOOP:
@@ -375,6 +377,7 @@ static int css_interpret_ccw(SubchDev *sch, hwaddr ccw_addr)
 }
 break;
 }
+fprintf(stderr, "CH DBG: %s: ret=%d\n", __func__, ret);
 sch->last_cmd = ccw;
 sch->last_cmd_valid = true;
 if (ret == 0) {


> > Which qemu version is this, btw.?
> >
> 
> git from yesterday.

Hm. Might be worth trying the s390-ccw-virtio-2.4 machine instead.

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH/RFC 0/4] dma ops and virtio

[Cornelia Huck]

On Thu, 29 Oct 2015 15:50:38 -0700
Andy Lutomirski  wrote:

> Progress!  After getting that sort-of-working, I figured out what was
> wrong with my earlier command, and I got that working, too.  Now I
> get:
> 
> qemu-system-s390x -fsdev
> local,id=virtfs1,path=/,security_model=none,readonly -device
> virtio-9p-ccw,fsdev=virtfs1,mount_tag=/dev/root -M s390-ccw-virtio
> -nodefaults -device sclpconsole,chardev=console -parallel none -net
> none -echr 1 -serial none -chardev stdio,id=console,signal=off,mux=on
> -serial chardev:console -mon chardev=console -vga none -display none
> -kernel arch/s390/boot/bzImage -append
> 'init=/home/luto/devel/virtme/virtme/guest/virtme-init
> psmouse.proto=exps "virtme_stty_con=rows 24 cols 150 iutf8"
> TERM=xterm-256color rootfstype=9p
> rootflags=ro,version=9p2000.L,trans=virtio,access=any
> raid=noautodetect debug'

The commandline looks sane AFAICS.

(...)

> vrfy: device 0.0.: rc=0 pgroup=0 mpath=0 vpm=80
> virtio_ccw 0.0.: Failed to set online: -5
> 
> ^^^ bad news!

I'd like to see where in the onlining process this fails. Could you set
up qemu tracing for css_* and virtio_ccw_* (instructions in
qemu/docs/tracing.txt)?

Which qemu version is this, btw.?

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH/RFC 0/4] dma ops and virtio

[Andy Lutomirski]

On Fri, Oct 30, 2015 at 1:25 AM, Cornelia Huck  wrote:
> On Thu, 29 Oct 2015 15:50:38 -0700
> Andy Lutomirski  wrote:
>
>> Progress!  After getting that sort-of-working, I figured out what was
>> wrong with my earlier command, and I got that working, too.  Now I
>> get:
>>
>> qemu-system-s390x -fsdev
>> local,id=virtfs1,path=/,security_model=none,readonly -device
>> virtio-9p-ccw,fsdev=virtfs1,mount_tag=/dev/root -M s390-ccw-virtio
>> -nodefaults -device sclpconsole,chardev=console -parallel none -net
>> none -echr 1 -serial none -chardev stdio,id=console,signal=off,mux=on
>> -serial chardev:console -mon chardev=console -vga none -display none
>> -kernel arch/s390/boot/bzImage -append
>> 'init=/home/luto/devel/virtme/virtme/guest/virtme-init
>> psmouse.proto=exps "virtme_stty_con=rows 24 cols 150 iutf8"
>> TERM=xterm-256color rootfstype=9p
>> rootflags=ro,version=9p2000.L,trans=virtio,access=any
>> raid=noautodetect debug'
>
> The commandline looks sane AFAICS.
>
> (...)
>
>> vrfy: device 0.0.: rc=0 pgroup=0 mpath=0 vpm=80
>> virtio_ccw 0.0.: Failed to set online: -5
>>
>> ^^^ bad news!
>
> I'd like to see where in the onlining process this fails. Could you set
> up qemu tracing for css_* and virtio_ccw_* (instructions in
> qemu/docs/tracing.txt)?

I have a file called events that contains:

css_*
virtio_ccw_*

pointing -trace events= at it results in a trace- file that's 549
bytes long and contains nothing.  Are wildcards not as well-supported
as the docs suggest?

>
> Which qemu version is this, btw.?
>

git from yesterday.

--Andy



-- 
Andy Lutomirski
AMA Capital Management, LLC
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH/RFC 0/4] dma ops and virtio

[Andy Lutomirski]

On Wed, Oct 28, 2015 at 5:04 PM, Christian Borntraeger
 wrote:
> Am 29.10.2015 um 07:22 schrieb Andy Lutomirski:
>> On Tue, Oct 27, 2015 at 3:48 PM, Christian Borntraeger
>>  wrote:
>>> This is an RFC to check if I am on the right track.  There
>>> are some attempts to unify the dma ops (Christoph) as well
>>> as some attempts to make virtio use the dma API (Andy).
>>>
>>> At todays discussion at the kernel summit, we concluded that
>>> we want to use the same code on all platforms, whereever
>>> possible, so having a dummy dma_op might be the easiest
>>> solution to keep virtio-ccw as similar as possible to
>>> virtio-pci. Andy Lutomirski will rework his patchset to
>>> unconditionally use the dma ops.  We will also need a
>>> compatibility quirk for powerpc to bypass the iommu mappings
>>> on older QEMU version (which translates to all versions as
>>> of today) and per device, e.g.  via device tree.  Ben
>>> Herrenschmidt will look into that.
>>
>> The combination of these patches plus my series don't link for me
>> unless I enable PCI.  Would s390 need to select HAS_DMA from VIRTIO or
>> something similar?
>
> Well, actually this is a potential improvement for series. I could just
> make the noop dma ops default for _all_ devices unless it has a per
> device dma_ops (e.g. s390 pci) and the unconditionally set HAS_DMA.
>>
>> Also, is it possible to use QEMU to emulate an s390x?  Even just:
>>
>> qemu-system-s390x -M s390-ccw-virtio
>
> Yes, we have no interactive bios and if no boot device is there is bios
> will load a disabled wait, which will exit qemu.
>
> Make sure to compile your kernel for z900  (processor type) as qemu does not
> handle all things of newer processors.
> You can then do
> qemu-system-s390x -nographic -m 256 -kernel vmlinux -initrd 
>

Progress!  After getting that sort-of-working, I figured out what was
wrong with my earlier command, and I got that working, too.  Now I
get:

qemu-system-s390x -fsdev
local,id=virtfs1,path=/,security_model=none,readonly -device
virtio-9p-ccw,fsdev=virtfs1,mount_tag=/dev/root -M s390-ccw-virtio
-nodefaults -device sclpconsole,chardev=console -parallel none -net
none -echr 1 -serial none -chardev stdio,id=console,signal=off,mux=on
-serial chardev:console -mon chardev=console -vga none -display none
-kernel arch/s390/boot/bzImage -append
'init=/home/luto/devel/virtme/virtme/guest/virtme-init
psmouse.proto=exps "virtme_stty_con=rows 24 cols 150 iutf8"
TERM=xterm-256color rootfstype=9p
rootflags=ro,version=9p2000.L,trans=virtio,access=any
raid=noautodetect debug'
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Initializing cgroup subsys cpuacct
Linux version 4.3.0-rc7-00403-ga2b5cd810259-dirty
(l...@amaluto.corp.amacapital.net) (gcc version 5.1.1 20150618 (Red
Hat Cross 5.1.1-3) (GCC) ) #328 SMP Thu Oct 29 15:46:05 PDT 2015
setup: Linux is running under KVM in 64-bit mode
setup: Max memory size: 128MB
Zone ranges:
  DMA  [mem 0x-0x7fff]
  Normal   empty
Movable zone start for each node
Early memory node ranges
  node   0: [mem 0x-0x07ff]
Initmem setup node 0 [mem 0x-0x07ff]
On node 0 totalpages: 32768
  DMA zone: 512 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 32768 pages, LIFO batch:7
PERCPU: Embedded 466 pages/cpu @07605000 s1868032 r8192 d32512 u1908736
pcpu-alloc: s1868032 r8192 d32512 u1908736 alloc=466*4096
pcpu-alloc: [0] 0 [0] 1
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 32256
Kernel command line:
init=/home/luto/devel/virtme/virtme/guest/virtme-init
psmouse.proto=exps "virtme_stty_con=rows 24 cols 150 iutf8"
TERM=xterm-256color rootfstype=9p
rootflags=ro,version=9p2000.L,trans=virtio,access=any
raid=noautodetect debug
PID hash table entries: 512 (order: 0, 4096 bytes)
Dentry cache hash table entries: 16384 (order: 5, 131072 bytes)
Inode-cache hash table entries: 8192 (order: 4, 65536 bytes)
Memory: 92552K/131072K available (8229K kernel code, 798K rwdata,
3856K rodata, 2384K init, 14382K bss, 38520K reserved, 0K
cma-reserved)
Write protected kernel read-only data: 0x10 - 0xccdfff
SLUB: HWalign=256, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
Running RCU self tests
Hierarchical RCU implementation.
RCU debugfs-based tracing is enabled.
RCU lockdep checking is enabled.
Build-time adjustment of leaf fanout to 64.
RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=2.
RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=2
NR_IRQS:3
clocksource: tod: mask: 0x max_cycles:
0x3b0a9be803b0a9, max_idle_ns: 1805497147909793 ns
console [ttyS1] enabled
Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
... MAX_LOCKDEP_SUBCLASSES:  8
... MAX_LOCK_DEPTH:  48
... MAX_LOCKDEP_KEYS:8191
... CLASSHASH_SIZE:  4096
... MAX_LOCKDEP_ENTRIES: 32768
... MAX_LOCKDEP_CHAINS:  

Re: [PATCH/RFC 0/4] dma ops and virtio

[Andy Lutomirski]

On Tue, Oct 27, 2015 at 3:48 PM, Christian Borntraeger
 wrote:
> This is an RFC to check if I am on the right track.  There
> are some attempts to unify the dma ops (Christoph) as well
> as some attempts to make virtio use the dma API (Andy).
>
> At todays discussion at the kernel summit, we concluded that
> we want to use the same code on all platforms, whereever
> possible, so having a dummy dma_op might be the easiest
> solution to keep virtio-ccw as similar as possible to
> virtio-pci. Andy Lutomirski will rework his patchset to
> unconditionally use the dma ops.  We will also need a
> compatibility quirk for powerpc to bypass the iommu mappings
> on older QEMU version (which translates to all versions as
> of today) and per device, e.g.  via device tree.  Ben
> Herrenschmidt will look into that.

The combination of these patches plus my series don't link for me
unless I enable PCI.  Would s390 need to select HAS_DMA from VIRTIO or
something similar?

Also, is it possible to use QEMU to emulate an s390x?  Even just:

qemu-system-s390x -M s390-ccw-virtio

exits immediately.  (I build from QEMU git today.)  I don't know what
options to pass to get -M s390 to do anything useful, and AFAICT it's
considered deprecated.

Help?

--Andy
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH/RFC 0/4] dma ops and virtio

[Christian Borntraeger]

Am 29.10.2015 um 07:22 schrieb Andy Lutomirski:
> On Tue, Oct 27, 2015 at 3:48 PM, Christian Borntraeger
>  wrote:
>> This is an RFC to check if I am on the right track.  There
>> are some attempts to unify the dma ops (Christoph) as well
>> as some attempts to make virtio use the dma API (Andy).
>>
>> At todays discussion at the kernel summit, we concluded that
>> we want to use the same code on all platforms, whereever
>> possible, so having a dummy dma_op might be the easiest
>> solution to keep virtio-ccw as similar as possible to
>> virtio-pci. Andy Lutomirski will rework his patchset to
>> unconditionally use the dma ops.  We will also need a
>> compatibility quirk for powerpc to bypass the iommu mappings
>> on older QEMU version (which translates to all versions as
>> of today) and per device, e.g.  via device tree.  Ben
>> Herrenschmidt will look into that.
> 
> The combination of these patches plus my series don't link for me
> unless I enable PCI.  Would s390 need to select HAS_DMA from VIRTIO or
> something similar?

Well, actually this is a potential improvement for series. I could just 
make the noop dma ops default for _all_ devices unless it has a per 
device dma_ops (e.g. s390 pci) and the unconditionally set HAS_DMA.
> 
> Also, is it possible to use QEMU to emulate an s390x?  Even just:
> 
> qemu-system-s390x -M s390-ccw-virtio

Yes, we have no interactive bios and if no boot device is there is bios
will load a disabled wait, which will exit qemu.

Make sure to compile your kernel for z900  (processor type) as qemu does not
handle all things of newer processors.
You can then do 
qemu-system-s390x -nographic -m 256 -kernel vmlinux -initrd 

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH/RFC 0/4] dma ops and virtio

[Joerg Roedel]

Hi Christian,

On Tue, Oct 27, 2015 at 11:48:47PM +0100, Christian Borntraeger wrote:
> Here is a very quick (still untested) shot at providing the s390 part:
> - patch1: dummy dma ops, inspired by the alpha code
> - patch2: replace some of the alpha functions with the dummy ones
> - patch3: allow per device dma ops for s390
> - patch4: wire up virtio dma ops

Thanks for the patches, I think they are a good start. I sent you
comments for two of the patches, the others look good to me.


Joerg

--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html