Nadav,
I agree that I should focus on getting VMBUS driver ported and making
OSv boot on hyper/V with ramdisk.
Also with yours and Avi's help I got most of the compilation errors
fixed ( I am down to under 150 lines of errors :-).
Anyway there are still some outstanding questions/issues I need help
with:
* The biggest issue is related to setting up VMBUS interrupt
handler which does not seem to be compatible with how OSv drivers
do it; I wonder if that part of freebsd code needs to converted
to some OSv constructs like interrupt subclasses
(processor::inter_processor_interrupt,
processor::gsi_edge_interrupt or processor::gsi_level_interrupt);
am I on the right track?; Here are the links to the relevant part
of the code
o
https://github.com/wkozaczuk/osv/blob/b97c056b464e28a4b8f1ff68c4b947109c08f20e/bsd/sys/dev/hyperv/vmbus/vmbus.cc#L128
o
https://github.com/wkozaczuk/osv/blob/b97c056b464e28a4b8f1ff68c4b947109c08f20e/bsd/sys/dev/hyperv/vmbus/vmbus.cc#L875-L924
o interrupt handle part1 in assembly code
https://github.com/wkozaczuk/osv/blob/b97c056b464e28a4b8f1ff68c4b947109c08f20e/bsd/sys/dev/hyperv/vmbus/amd64/vmbus_vector.S#L34-L46
o interrupt handler part 2 -
https://github.com/wkozaczuk/osv/blob/b97c056b464e28a4b8f1ff68c4b947109c08f20e/bsd/sys/dev/hyperv/vmbus/vmbus.cc#L667-L689
* pause_sbt does not exist - I wonder what the equivalent in OSv is
-
https://github.com/freebsd/freebsd/blob/7a5edbcf70144666ddbbd33f6890fe2ad265ff09/sys/kern/kern_synch.c#L301-L334
* what would be the equivalent of - callout_reset_sbt_curcpu
(https://github.com/freebsd/freebsd/blob/7a5edbcf70144666ddbbd33f6890fe2ad265ff09/sys/sys/callout.h#L107)
* Should I port taskqueue_create_fast() and
taskqueue_start_threads_cpuset() from freebsd add them to the set
of similar functions in bsd/sys/sys/taskqueue.h?
* lapic_ipi_free and lapic_ipi_alloc doe not exist in OSv but I
think they are used to allocate/free interrupt handlers in OSv so
somehow in OSv it would have to be done differently
Most of the relevant code that has the issues I am describing above
is in these 3 files (all other files in vmbus subdirectory compile at
least):
*
https://github.com/wkozaczuk/osv/blob/hyper_v/bsd/sys/dev/hyperv/vmbus/vmbus.cc
*
https://github.com/wkozaczuk/osv/blob/hyper_v/bsd/sys/dev/hyperv/vmbus/vmbus_chan.cc
*
https://github.com/wkozaczuk/osv/blob/hyper_v/bsd/sys/dev/hyperv/vmbus/amd64/vmbus_vector.S
Thanks in advance for your help,
Waldek
On Thursday, April 27, 2017 at 5:07:22 AM UTC-4, Nadav Har'El wrote:
On Thu, Apr 27, 2017 at 11:37 AM, Avi Kivity <a...@scylladb.com
<javascript:>> wrote:
On 04/27/2017 07:37 AM, Waldek Kozaczuk wrote:
Here are other issues where I am seeking advice:
* *SYSCTL* in FreeBSD provides configuration management
mechanism for kernel and device drivers
(https://www.freebsd.org/cgi/man.cgi?sysctl(8))
<https://www.freebsd.org/cgi/man.cgi?sysctl%288%29%29>.
As I understand OSv does not support similar mechanism
(and possibly will never do). Does it mean that all
sysctl related code should be disabled in the Hyper/V
drivers that I am bringing in from FreeBSD?
Yes, it can be ignored, at least now.
If you're importing code and don't want to bother porting pieces
of it yet, please keep those parts #ifdef'ed out, to make it
easier to re-enabling them in the future.
* The*vmbus* driver uses 4 atomic_* functions that are not
present in OSv; should they be brought in from BSD
(https://github.com/freebsd/freebsd/blob/master/sys/amd64/include/atomic.h
<https://github.com/freebsd/freebsd/blob/master/sys/amd64/include/atomic.h>);
it looks like some of these functions are intended to be
used by kernel modules - does it have any implications
to OSv?
o atomic_swap_long
o atomic_testandclear_int
o atomic_testandclear_long
o atomic_testandset_int
You can just bring them over (or implement them with
std::atomic).
In bsd/x64/machine/atomic.h we already have many of the BSD
atomic_functions, you can add more there (and can probably copy
them from BSD, I assume that's what we did with the stuff already
there). It's easier to copy than re-implement because then you
won't need to think hard about what are the specific guarantees
that these functions are supposed to make.
And like Avi already said, there's also C++'s std::atomic which
you can use in new code, but I think it will be easier for you
not to have to modify that code.
* In some places the code uses DELAY() macro (example
https://github.com/freebsd/freebsd/blob/becaf5f7f2e608c98d9c470f1ae004b24882f6d5fe3e20e5/sys/dev/hyperv/vmbus/vmbus_chan.c#L687
<https://github.com/freebsd/freebsd/blob/becaf5f7f2e608c98d9c470f1ae004b5fe3e20e5/sys/dev/hyperv/vmbus/vmbus_chan.c#L687>)
that I believe makes it wait/sleep on the current thread
(https://www.freebsd.org/cgi/man.cgi?query=DELAY&apropos=0&sektion=9
<https://www.freebsd.org/cgi/man.cgi?query=DELAY&apropos=0&sektion=9>).
How should it be converted to OSv?
There is sched::thread::sleep().
According to delay(9) (see
http://www.unix.com/man-page/freebsd/9/delay/
<http://www.unix.com/man-page/freebsd/9/delay/>), delay() is
supposed to do a busy loop for the given number of microseconds.
So DELAY(1000) (as in that code) should busy loop until a whole
millisecond has passed. I agree with Avi that when such a long
amount of time is concerned, it makes sense to just use
sched::thread::sleep() or even usleep() which already takes the
correct units - and let other threads work. For shorter periods,
you can easily implement DELAY, as a busy loop which continuously
calls gettimeofday() and stops looping when enough microseconds
have passed (or it would be even nicer to implement the same
thing using C++'s std::chrono mechanisms).
* In some places the code uses __compiler_membar() macro
(https://github.com/freebsd/freebsd/blob/becaf5f7f2e608c98d9c470f1ae004b5fe3e20e5/sys/sys/cdefs.h#L108
<https://github.com/freebsd/freebsd/blob/becaf5f7f2e608c98d9c470f1ae004b5fe3e20e5/sys/sys/cdefs.h#L108>).
How should it be converted to OSv?
std::atomic_signal_fence(std::memory_order_seq_cst)
We also have <osv/barrier.hh> with the barrier() function, but I
agree that std::atomic_signal_fence looks more standard.
* In some places the code uses
critical_enter()/citical_exit()
(https://www.freebsd.org/cgi/man.cgi?query=critical_enter&sektion=9
<https://www.freebsd.org/cgi/man.cgi?query=critical_enter&sektion=9>)
to prevent thread from migrating to other CPU. How
should it be converted to OSv?
WITH_LOCK(preempt_lock) { .... }
If *really* the only concern is migration to other CPUs, we also
have WITH_LOCK(migration_lock).
But looking at the critical(9) manual page, it seems indeed you
should indeed use preempt_lock because the manual has the text
"These functions are used to prevent preemption".
* In some places the code uses VM_GUEST_VM macro and
vm_guest global variable (example
https://github.com/freebsd/freebsd/blob/becaf5f7f2e608c98d9c470f1ae004b5fe3e20e5/sys/dev/hyperv/vmbus/hyperv.c#L147-L148
<https://github.com/freebsd/freebsd/blob/becaf5f7f2e608c98d9c470f1ae004b5fe3e20e5/sys/dev/hyperv/vmbus/hyperv.c#L147-L148>)
to determine if code is running on HyperV? What would be
the best way to convert it to OSv? Or maybe it does not
apply as in OSv hypervisor is detected during boot time
and corresponding code would not be executed anyway?
OSv also has some code which can tell a curious application on
which hypervisor is running (see
http://www.unix.com/man-page/freebsd/9/delay/
<http://www.unix.com/man-page/freebsd/9/delay/>), but I assume
that most of the HyperV-specific code would only get run when
HyperV is detected, so that code wouldn't need to check if hyperv
is available. I'm not familiar with that code to give a better
answer.
* The struct*ifqueue* (sys/net/ifq.h) has following changes
which affect porting netvsc driver:
o *new field ifq_drv_maxlen* -
https://github.com/freebsd/freebsd/blob/becaf5f7f2e608c98d9c470f1ae004b5fe3e20e5/sys/dev/hyperv/netvsc/if_hn.c#L1348
<https://github.com/freebsd/freebsd/blob/becaf5f7f2e608c98d9c470f1ae004b5fe3e20e5/sys/dev/hyperv/netvsc/if_hn.c#L1348>
o removed field *altq_flags* which is used by macro
IFQ_SET_READY
(https://github.com/freebsd/freebsd/blob/8b6efb8965fa7f80abdc363663ed6dafa0c70ecd/sys/net/ifq.h#L228
<https://github.com/freebsd/freebsd/blob/8b6efb8965fa7f80abdc363663ed6dafa0c70ecd/sys/net/ifq.h#L228>)
o Shall we simply comment out this part of the code as
nothing else in OSv current code uses it?
Here are the things I have changed so far:
* Fixed conversion errors by using static_cast<>
o I noticed that in other places (type *) construct was
used to achieve the same thing as static_cast<type *>();
which one is preferred?
* Replaced pause() with bsd_pause() (OSv version of the same
functionality I think - is this correct?)
* Replaced CSUM_* new constants with old ones (for example
CSUM_IP_TCP with CSUM_TCP) because of the changes in
https://github.com/freebsd/freebsd/blob/master/sys/sys/mbuf.h
<https://github.com/freebsd/freebsd/blob/master/sys/sys/mbuf.h>
* Due to the changes of *struct mbu*f layout in FreeBSD
(https://github.com/freebsd/freebsd/commit/08b6ceecbb05f0b7bd21e29feee1624747a5b172
<https://github.com/freebsd/freebsd/commit/08b6ceecbb05f0b7bd21e29feee1624747a5b172>)
I changed all relevant places in netvsc like in this example:
o (m)->m_pkthdr.len to (m)-M_dat.MH.MH__pkthdr.len
o Is this the right fix?
* Due to the difference of field if_snd in *struct ifnet*
between OSv and FreeBSD (look at
https://github.com/cloudius-systems/osv/commit/19f24526cdd9d937b8d33d25ed7c6437dd5ea89f
<https://github.com/cloudius-systems/osv/commit/19f24526cdd9d937b8d33d25ed7c6437dd5ea89f>)
which makes macros like IFQ_DRV_DEQUEUE not-compilable, I
had to replace IFQ_DRV_DEQUEUE macro with IF_DEQUEUE and
IFQ_DRV_PREPEND with IF_PREPEND
o I did it based on the reading of
https://www.freebsd.org/cgi/man.cgi?query=altq&sektion=9
<https://www.freebsd.org/cgi/man.cgi?query=altq&sektion=9>
which makes me think that driver managed queue are not
supported in OSv
o Please see my changes here -
https://github.com/wkozaczuk/osv/commit/e486143830fcd2020a267cba7f1d3a38df8c34d6#diff-4f9f901a3850cc1241e39ad0cf0d2cb0R4119
<https://github.com/wkozaczuk/osv/commit/e486143830fcd2020a267cba7f1d3a38df8c34d6#diff-4f9f901a3850cc1241e39ad0cf0d2cb0R4119>
* Added *boolean_t* which was missing in OSv
* Replaced *rdmsr*() and *wrmsr*() with processor::rdmsr() and
processor::wrmsr()
* Disabled PCI Pass Through support in netvsc driver which use
PCIB related macros like PCIB_MAP_MSI and functions pcib_
not present in OSv at this time
o For example this header is not present in OSv -
https://github.com/freebsd/freebsd/blob/master/sys/dev/pci/pcib_private.h
<https://github.com/freebsd/freebsd/blob/master/sys/dev/pci/pcib_private.h>
o Read more about CPI Pass Through here -
https://www.slideshare.net/iXsystems/pci-passthrough-freebsd-vm-on-hyperv-meetbsd-california-2016
<https://www.slideshare.net/iXsystems/pci-passthrough-freebsd-vm-on-hyperv-meetbsd-california-2016>
* Temporarily commented out usage of
mtx_lock(&Giant)/mtx_unlock(&Giant) in vmbus.cc
<http://vmbus.cc>
o Look for example here -
https://github.com/freebsd/freebsd/blob/8b6efb8965fa7f80abdc363663ed6dafa0c70ecd/sys/dev/hyperv/vmbus/vmbus.c#L520-L522
<https://github.com/freebsd/freebsd/blob/8b6efb8965fa7f80abdc363663ed6dafa0c70ecd/sys/dev/hyperv/vmbus/vmbus.c#L520-L522>
o What would be the best way to achieve similar locking
functionality in OSv? Should corresponding logic use
some kind of global lock but only limited to hyperv vmbus?
* Added ift_counter enum and if_inc_counter() function to if_var.h
* Added bsd/sys/net/rndis.h
* Added bsd/sys/sys/timetc.h
* Added bsd/sys/sys/buf_ring.h
There are other devices that I am not sure if they need to be
ported to OSv:
* kbd - synthetic keyboard (most likely not needed)
* pcib - synthetic PCI bridge driver
(https://reviews.freebsd.org/D8332
<https://reviews.freebsd.org/D8332>)
* kvp - key-value pair driver
* heartbeat
* *timesync* - I was wondering if this one needs to be ported
to possibly synchronize clock with the hypervisor; excuse my
ignorance in this subject but maybe OSv uses some generic
mechanism to synchronize clock that works on all hypervisors?
I did not see any console driver. Is it the case that there is
some standard mechanism to use console that works on all
hypervisors?
I am also thinking of changing my original plan. Instead of
porting all drivers first I will first port vmbus only and then
work on implementing hyperv::vmbus and hyperv::hyperv_driver
classes to make OSv at least boot on Hyper/V and print something
to the console.
Also if anybody wants to collaborate I created the branch here -
https://github.com/wkozaczuk/osv/tree/hyper_v
<https://github.com/wkozaczuk/osv/tree/hyper_v>. It is pretty
much in flux and most hyperv files do not compile yet. I would
definitely need help port storvsc and netvsc drivers.
Also I am apologize for such a long email but I wanted to list
all my specific questions to get a specific help/feedback.
Cheers and thanks in advance,
Waldek
--
You received this message because you are subscribed to the
Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to osv-dev+u...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the
Google Groups "OSv Development" group.
To unsubscribe from this group and stop receiving emails from
it, send an email to osv-dev+u...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
