Re: [RFC PATCH 00/28] Linux Kernel Library

[Octavian Purdila]

On Sat, Nov 7, 2015 at 12:48 PM, Richard W.M. Jones  wrote:
>
> I just pushed a (very early) WIP branch that contains changes to
> libguestfs to add an LKL backend:
>
>   https://github.com/rwmjones/libguestfs/tree/lkl
>
> Read the README file in the libguestfs sources before starting,
> followed by the instructions in the commit message:
>
>   
> https://github.com/rwmjones/libguestfs/commit/e38525f0b984d0a426f3348d95f2033673d4eaa4
>

Hi Richard,

Thanks for this, I was able to replicate it locally.

I started working on adding a new library, liblkl-redirect.a, that
redefines common libc symbols to use lkl. We then can link the daemon
with this lib and lkl to avoid ugly ifdefs in the daemon code.

Thanks,
Tavi
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Octavian Purdila]

On Sun, Nov 8, 2015 at 4:42 PM, yalin wang  wrote:
>
> On Nov 4, 2015, at 07:06, Octavian Purdila 
> wrote:
>
> On Tue, Nov 3, 2015 at 11:40 PM, Richard Weinberger
>  wrote:
>
> Hi Richard,
>
> On Tue, Nov 3, 2015 at 9:20 PM, Octavian Purdila
>  wrote:
>
> LKL (Linux Kernel Library) is aiming to allow reusing the Linux kernel code
> as extensively as possible with minimal effort and reduced maintenance
> overhead.
>
> Examples of how LKL can be used are: creating userspace applications
> (running on Linux and other operating systems) that can read or write Linux
> filesystems or can use the Linux networking stack, creating kernel drivers
> for other operating systems that can read Linux filesystems, bootloaders
> support for reading/writing Linux filesystems, etc.
>
> With LKL, the kernel code is compiled into an object file that can be
> directly linked by applications. The API offered by LKL is based on the
> Linux system call interface.
>
> LKL is implemented as an architecture port in arch/lkl. It relies on host
> operations defined by the application or a host library (tools/lkl/lib).
>
> The latest LKL version can be found at g...@github.com:lkl/linux.git
>
>
> Or more copy friendly: https://github.com/lkl/linux.git
>
> FAQ
> ===
>
> Q: How is LKL different from UML?
> A: UML provides a full OS environment (e.g. user/kernel separation, user
> processes) and also has requirements (a filesystem, processes, etc.) that
> makes it hard to use it for standalone applications. UML also relies
> heavily on Linux hosts. On the other hand LKL is designed to be linked
> directly with the application and hence does not have user/kernel
> separation which makes it easier to use it in standalone applications.
>
>
> So, this is a "liblinux" where applications are directly linked
> against the kernel.
> IOW system calls are plain function calls into the kernel?
>
>
> More like "thread" calls. All system calls are executed in a dedicate
> (kernel) thread to avoid race conditions with the "interrupt" path.
>
> why not call sys_XXX()  function directly?
> since kernel have implement lots of spin_locks to avoid race with normal
> path
> in IRQ handle ,  isn’t  it ?
> for example, you timer IRQ can be simulated by SIGALARM signal,
> and the signal handler can check if IRQ is disabled ,
> if not , then continuing , otherwise , return directly ..
> it is not safe ?
>

Hi Yalin,

We need to have a proper Linux context in order to issue system calls
(e.g. current needs to point to a proper Linux kernel thread_struct).
We also want to let the Linux scheduler to select what kernel threads
run at a given time. Lets say that currently a ksoftirqd runs and the
application want to issue a system call. In this case we want to wait
for ksoftirqd to complete then run the system call.

The simplest way I found to do that is to have the system calls
execute from Linux kernel threads hence the need to queue the system
calls from the application thread to the Linux kernel system call
thread.

(BTW, we can have multiple system call threads if needed and the 2.6
implementation has that, but in order to simplify the review process I
decided to throw away that for the moment).

Thanks,
Tavi
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Octavian Purdila]

On Sun, Nov 8, 2015 at 4:42 PM, yalin wang  wrote:
>
> On Nov 4, 2015, at 07:06, Octavian Purdila 
> wrote:
>
> On Tue, Nov 3, 2015 at 11:40 PM, Richard Weinberger
>  wrote:
>
> Hi Richard,
>
> On Tue, Nov 3, 2015 at 9:20 PM, Octavian Purdila
>  wrote:
>
> LKL (Linux Kernel Library) is aiming to allow reusing the Linux kernel code
> as extensively as possible with minimal effort and reduced maintenance
> overhead.
>
> Examples of how LKL can be used are: creating userspace applications
> (running on Linux and other operating systems) that can read or write Linux
> filesystems or can use the Linux networking stack, creating kernel drivers
> for other operating systems that can read Linux filesystems, bootloaders
> support for reading/writing Linux filesystems, etc.
>
> With LKL, the kernel code is compiled into an object file that can be
> directly linked by applications. The API offered by LKL is based on the
> Linux system call interface.
>
> LKL is implemented as an architecture port in arch/lkl. It relies on host
> operations defined by the application or a host library (tools/lkl/lib).
>
> The latest LKL version can be found at g...@github.com:lkl/linux.git
>
>
> Or more copy friendly: https://github.com/lkl/linux.git
>
> FAQ
> ===
>
> Q: How is LKL different from UML?
> A: UML provides a full OS environment (e.g. user/kernel separation, user
> processes) and also has requirements (a filesystem, processes, etc.) that
> makes it hard to use it for standalone applications. UML also relies
> heavily on Linux hosts. On the other hand LKL is designed to be linked
> directly with the application and hence does not have user/kernel
> separation which makes it easier to use it in standalone applications.
>
>
> So, this is a "liblinux" where applications are directly linked
> against the kernel.
> IOW system calls are plain function calls into the kernel?
>
>
> More like "thread" calls. All system calls are executed in a dedicate
> (kernel) thread to avoid race conditions with the "interrupt" path.
>
> why not call sys_XXX()  function directly?
> since kernel have implement lots of spin_locks to avoid race with normal
> path
> in IRQ handle ,  isn’t  it ?
> for example, you timer IRQ can be simulated by SIGALARM signal,
> and the signal handler can check if IRQ is disabled ,
> if not , then continuing , otherwise , return directly ..
> it is not safe ?
>

Hi Yalin,

We need to have a proper Linux context in order to issue system calls
(e.g. current needs to point to a proper Linux kernel thread_struct).
We also want to let the Linux scheduler to select what kernel threads
run at a given time. Lets say that currently a ksoftirqd runs and the
application want to issue a system call. In this case we want to wait
for ksoftirqd to complete then run the system call.

The simplest way I found to do that is to have the system calls
execute from Linux kernel threads hence the need to queue the system
calls from the application thread to the Linux kernel system call
thread.

(BTW, we can have multiple system call threads if needed and the 2.6
implementation has that, but in order to simplify the review process I
decided to throw away that for the moment).

Thanks,
Tavi
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Octavian Purdila]

On Sat, Nov 7, 2015 at 12:48 PM, Richard W.M. Jones  wrote:
>
> I just pushed a (very early) WIP branch that contains changes to
> libguestfs to add an LKL backend:
>
>   https://github.com/rwmjones/libguestfs/tree/lkl
>
> Read the README file in the libguestfs sources before starting,
> followed by the instructions in the commit message:
>
>   
> https://github.com/rwmjones/libguestfs/commit/e38525f0b984d0a426f3348d95f2033673d4eaa4
>

Hi Richard,

Thanks for this, I was able to replicate it locally.

I started working on adding a new library, liblkl-redirect.a, that
redefines common libc symbols to use lkl. We then can link the daemon
with this lib and lkl to avoid ugly ifdefs in the daemon code.

Thanks,
Tavi
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Hajime Tazaki]

Hello Octavian,

At Tue,  3 Nov 2015 22:20:31 +0200,
Octavian Purdila wrote:
> 
> 
> Q: How is LKL different from LibOS?
> A: LibOS re-implements high-level kernel APIs for timers, softirqs,
> scheduling, sysctl, SLAB/SLUB, etc. LKL behaves like any arch port,
> implementing the arch level operations requested by the Linux kernel. LKL
> also offers a host interface so that support for multiple hosts can be
> easily implemented.

I review most of code with the help of document and paper (2010).

I think LKL and LibOS are essentially the same thing.

I describe the current differences of both features, which I
believe there are no fundamental ones (i.e., both can
improve by putting some efforts).

- LKL
 (beautiful) arch implementation (I like it)
 fully kbuild compliant
 rich fs support
 host support: POSIX, win, haiku, etc

- LibOS
 existing application integration
 (semi-automated) system call table generation
 multiple process support (via system call proxy)
 various network backends (raw socket, DPDK, netmap, tap)
 symbol namespace separation
 host support: == rump hypercall (POSIX, xen,
   qemu/kvm/baremetal(under development)), ns-3 simulator

# I can't find network support within the current patch but
  there is/will be a certain code that LKL can play with
  networking subsystem.

existing application integration is really important when
you want to configure network stack: since the configuration
of file systems is just a mount(), but configurations of
network stack need much userspace applications like iproute2
(ip, ss, tc) etc, which is not trivial to re-implement.


-- Hajime
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Hajime Tazaki]

Hello Octavian,

At Tue,  3 Nov 2015 22:20:31 +0200,
Octavian Purdila wrote:
> 
> 
> Q: How is LKL different from LibOS?
> A: LibOS re-implements high-level kernel APIs for timers, softirqs,
> scheduling, sysctl, SLAB/SLUB, etc. LKL behaves like any arch port,
> implementing the arch level operations requested by the Linux kernel. LKL
> also offers a host interface so that support for multiple hosts can be
> easily implemented.

I review most of code with the help of document and paper (2010).

I think LKL and LibOS are essentially the same thing.

I describe the current differences of both features, which I
believe there are no fundamental ones (i.e., both can
improve by putting some efforts).

- LKL
 (beautiful) arch implementation (I like it)
 fully kbuild compliant
 rich fs support
 host support: POSIX, win, haiku, etc

- LibOS
 existing application integration
 (semi-automated) system call table generation
 multiple process support (via system call proxy)
 various network backends (raw socket, DPDK, netmap, tap)
 symbol namespace separation
 host support: == rump hypercall (POSIX, xen,
   qemu/kvm/baremetal(under development)), ns-3 simulator

# I can't find network support within the current patch but
  there is/will be a certain code that LKL can play with
  networking subsystem.

existing application integration is really important when
you want to configure network stack: since the configuration
of file systems is just a mount(), but configurations of
network stack need much userspace applications like iproute2
(ip, ss, tc) etc, which is not trivial to re-implement.


-- Hajime
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Octavian Purdila]

On Sat, Nov 7, 2015 at 2:35 AM, Richard Weinberger  wrote:
> Am 04.11.2015 um 15:15 schrieb Octavian Purdila:
>> We could redefine the syscalls/libc symbols to call lkl_sys_ functions
>> in launch-lkl, e.g.:
>>
>> int opendir(const char *path)
>> {
>>return lkl_opendir(new_path)
>> }
>
> To get a better feeling how LKL behaves I've started with a tool
> to mount any Linux filesystem by FUSE.
> I.e. such that we can finally automount without root and bugs in filesystem
> code won't hurt that much.
>

Hi Richard,

I finished coding mine up yesterday :) I've just pushed it to github
if you want to take a look at it (together with some small fixes).

> lkl_sys_fstatat64() uses the type struct lkl_stat64. Where is it defined?
> git grep is unable to locate it.
> At least it seems to be incompatible with my local struct stat.
>

When doing the build in tools/lkl the LKL headers are installed in
include/lkl. You should find it in there:

$:~/src/linux/tools/lkl$ find include/ -type f | xargs grep lkl_stat64
include/lkl/asm-generic/stat.h:struct lkl_stat64 {

> And why is there no lkl_sys_openat() syscall?
>

Didn't get to test it hence I didn't add it to the API yet, as many
other system calls.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Octavian Purdila]

On Sat, Nov 7, 2015 at 2:35 AM, Richard Weinberger  wrote:
> Am 04.11.2015 um 15:15 schrieb Octavian Purdila:
>> We could redefine the syscalls/libc symbols to call lkl_sys_ functions
>> in launch-lkl, e.g.:
>>
>> int opendir(const char *path)
>> {
>>return lkl_opendir(new_path)
>> }
>
> To get a better feeling how LKL behaves I've started with a tool
> to mount any Linux filesystem by FUSE.
> I.e. such that we can finally automount without root and bugs in filesystem
> code won't hurt that much.
>
> lkl_sys_fstatat64() uses the type struct lkl_stat64. Where is it defined?
> git grep is unable to locate it.
> At least it seems to be incompatible with my local struct stat.
>
> And why is there no lkl_sys_openat() syscall?
>
> Thanks,
> //richard
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Richard W.M. Jones]

I just pushed a (very early) WIP branch that contains changes to
libguestfs to add an LKL backend:

  https://github.com/rwmjones/libguestfs/tree/lkl

Read the README file in the libguestfs sources before starting,
followed by the instructions in the commit message:

  
https://github.com/rwmjones/libguestfs/commit/e38525f0b984d0a426f3348d95f2033673d4eaa4

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Richard W.M. Jones]

I just pushed a (very early) WIP branch that contains changes to
libguestfs to add an LKL backend:

  https://github.com/rwmjones/libguestfs/tree/lkl

Read the README file in the libguestfs sources before starting,
followed by the instructions in the commit message:

  
https://github.com/rwmjones/libguestfs/commit/e38525f0b984d0a426f3348d95f2033673d4eaa4

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Octavian Purdila]

On Sat, Nov 7, 2015 at 2:35 AM, Richard Weinberger  wrote:
> Am 04.11.2015 um 15:15 schrieb Octavian Purdila:
>> We could redefine the syscalls/libc symbols to call lkl_sys_ functions
>> in launch-lkl, e.g.:
>>
>> int opendir(const char *path)
>> {
>>return lkl_opendir(new_path)
>> }
>
> To get a better feeling how LKL behaves I've started with a tool
> to mount any Linux filesystem by FUSE.
> I.e. such that we can finally automount without root and bugs in filesystem
> code won't hurt that much.
>
> lkl_sys_fstatat64() uses the type struct lkl_stat64. Where is it defined?
> git grep is unable to locate it.
> At least it seems to be incompatible with my local struct stat.
>
> And why is there no lkl_sys_openat() syscall?
>
> Thanks,
> //richard
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Octavian Purdila]

On Sat, Nov 7, 2015 at 2:35 AM, Richard Weinberger  wrote:
> Am 04.11.2015 um 15:15 schrieb Octavian Purdila:
>> We could redefine the syscalls/libc symbols to call lkl_sys_ functions
>> in launch-lkl, e.g.:
>>
>> int opendir(const char *path)
>> {
>>return lkl_opendir(new_path)
>> }
>
> To get a better feeling how LKL behaves I've started with a tool
> to mount any Linux filesystem by FUSE.
> I.e. such that we can finally automount without root and bugs in filesystem
> code won't hurt that much.
>

Hi Richard,

I finished coding mine up yesterday :) I've just pushed it to github
if you want to take a look at it (together with some small fixes).

> lkl_sys_fstatat64() uses the type struct lkl_stat64. Where is it defined?
> git grep is unable to locate it.
> At least it seems to be incompatible with my local struct stat.
>

When doing the build in tools/lkl the LKL headers are installed in
include/lkl. You should find it in there:

$:~/src/linux/tools/lkl$ find include/ -type f | xargs grep lkl_stat64
include/lkl/asm-generic/stat.h:struct lkl_stat64 {

> And why is there no lkl_sys_openat() syscall?
>

Didn't get to test it hence I didn't add it to the API yet, as many
other system calls.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Richard W.M. Jones]

On Sat, Nov 07, 2015 at 01:35:36AM +0100, Richard Weinberger wrote:
> Am 04.11.2015 um 15:15 schrieb Octavian Purdila:
> > We could redefine the syscalls/libc symbols to call lkl_sys_ functions
> > in launch-lkl, e.g.:
> > 
> > int opendir(const char *path)
> > {
> >return lkl_opendir(new_path)
> > }
> 
> To get a better feeling how LKL behaves I've started with a tool
> to mount any Linux filesystem by FUSE.
> I.e. such that we can finally automount without root and bugs in filesystem
> code won't hurt that much.

guestmount already does this:

http://libguestfs.org/guestmount.1.html

By porting a small amount of code from the daemon/ directory, it could
do it using lkl too.  See:

http://www.gossamer-threads.com/lists/linux/kernel/2296116#2296116

Rich.

> lkl_sys_fstatat64() uses the type struct lkl_stat64. Where is it defined?
> git grep is unable to locate it.
> At least it seems to be incompatible with my local struct stat.
> 
> And why is there no lkl_sys_openat() syscall?
> 
> Thanks,
> //richard

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Richard Weinberger]

Am 04.11.2015 um 15:15 schrieb Octavian Purdila:
> We could redefine the syscalls/libc symbols to call lkl_sys_ functions
> in launch-lkl, e.g.:
> 
> int opendir(const char *path)
> {
>return lkl_opendir(new_path)
> }

To get a better feeling how LKL behaves I've started with a tool
to mount any Linux filesystem by FUSE.
I.e. such that we can finally automount without root and bugs in filesystem
code won't hurt that much.

lkl_sys_fstatat64() uses the type struct lkl_stat64. Where is it defined?
git grep is unable to locate it.
At least it seems to be incompatible with my local struct stat.

And why is there no lkl_sys_openat() syscall?

Thanks,
//richard
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Richard Weinberger]

Am 04.11.2015 um 15:15 schrieb Octavian Purdila:
> We could redefine the syscalls/libc symbols to call lkl_sys_ functions
> in launch-lkl, e.g.:
> 
> int opendir(const char *path)
> {
>return lkl_opendir(new_path)
> }

To get a better feeling how LKL behaves I've started with a tool
to mount any Linux filesystem by FUSE.
I.e. such that we can finally automount without root and bugs in filesystem
code won't hurt that much.

lkl_sys_fstatat64() uses the type struct lkl_stat64. Where is it defined?
git grep is unable to locate it.
At least it seems to be incompatible with my local struct stat.

And why is there no lkl_sys_openat() syscall?

Thanks,
//richard
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Richard W.M. Jones]

On Sat, Nov 07, 2015 at 01:35:36AM +0100, Richard Weinberger wrote:
> Am 04.11.2015 um 15:15 schrieb Octavian Purdila:
> > We could redefine the syscalls/libc symbols to call lkl_sys_ functions
> > in launch-lkl, e.g.:
> > 
> > int opendir(const char *path)
> > {
> >return lkl_opendir(new_path)
> > }
> 
> To get a better feeling how LKL behaves I've started with a tool
> to mount any Linux filesystem by FUSE.
> I.e. such that we can finally automount without root and bugs in filesystem
> code won't hurt that much.

guestmount already does this:

http://libguestfs.org/guestmount.1.html

By porting a small amount of code from the daemon/ directory, it could
do it using lkl too.  See:

http://www.gossamer-threads.com/lists/linux/kernel/2296116#2296116

Rich.

> lkl_sys_fstatat64() uses the type struct lkl_stat64. Where is it defined?
> git grep is unable to locate it.
> At least it seems to be incompatible with my local struct stat.
> 
> And why is there no lkl_sys_openat() syscall?
> 
> Thanks,
> //richard

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Octavian Purdila]

On Wed, Nov 4, 2015 at 3:50 PM, Richard W.M. Jones  wrote:
> On Wed, Nov 04, 2015 at 01:24:03AM +0200, Octavian Purdila wrote:
>> Thanks for the pointers Richard, I am going to take a look at it.
>
> Now I've had a chance to look at some of the example LKL tools, here's
> what this actually involves.  It's not actually a great deal of work,
> it could probably be done in a day or two, but see my question about
> `lkl_sys_*' below.
>
> libguestfs (the library part) needs to talk over an RPC connection to
> its daemon.  See diagram here:
>
> http://libguestfs.org/guestfs-internals.1.html
>
> The code in src/launch-{direct,libvirt,uml,...}.c sets up that
> connection and runs the daemon -- normally inside a qemu wrapper, but
> it could be inside UML.  For LKL I think it should just fork the
> daemon directly.
>
> The daemon would then be linked to LKL.
>
> So really what's needed is a src/launch-lkl.c probably modelled after
> one of these current backends:
>
> https://github.com/libguestfs/libguestfs/blob/master/src/launch-uml.c
> https://github.com/libguestfs/libguestfs/blob/master/src/launch-unix.c
>
> and then recompile the daemon to link to LKL:
>
> https://github.com/libguestfs/libguestfs/tree/master/daemon
>
> and pass the list of disk images to the daemon, probably best to do
> that on the guestfsd command line.
>
> My only problem here: you can't just link to daemon to LKL, do you
> have to change all of the system calls from `foo' to `lkl_sys_foo'?
> That's an awful lot of #ifdefs ...
>

We could redefine the syscalls/libc symbols to call lkl_sys_ functions
in launch-lkl, e.g.:

int opendir(const char *path)
{
   return lkl_opendir(new_path)
}
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Richard W.M. Jones]

On Wed, Nov 04, 2015 at 01:24:03AM +0200, Octavian Purdila wrote:
> Thanks for the pointers Richard, I am going to take a look at it.

Now I've had a chance to look at some of the example LKL tools, here's
what this actually involves.  It's not actually a great deal of work,
it could probably be done in a day or two, but see my question about
`lkl_sys_*' below.

libguestfs (the library part) needs to talk over an RPC connection to
its daemon.  See diagram here:

http://libguestfs.org/guestfs-internals.1.html

The code in src/launch-{direct,libvirt,uml,...}.c sets up that
connection and runs the daemon -- normally inside a qemu wrapper, but
it could be inside UML.  For LKL I think it should just fork the
daemon directly.

The daemon would then be linked to LKL.

So really what's needed is a src/launch-lkl.c probably modelled after
one of these current backends:

https://github.com/libguestfs/libguestfs/blob/master/src/launch-uml.c
https://github.com/libguestfs/libguestfs/blob/master/src/launch-unix.c

and then recompile the daemon to link to LKL:

https://github.com/libguestfs/libguestfs/tree/master/daemon

and pass the list of disk images to the daemon, probably best to do
that on the guestfsd command line.

My only problem here: you can't just link to daemon to LKL, do you
have to change all of the system calls from `foo' to `lkl_sys_foo'?
That's an awful lot of #ifdefs ...

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Austin S Hemmelgarn]

On 2015-11-03 18:24, Octavian Purdila wrote:

On Wed, Nov 4, 2015 at 12:45 AM, Richard W.M. Jones  wrote:

I'm dubious that a lib-based approach could support LVM, partioning,
ntfs-3g, qcow2, vmdk and all the other libguestfs stuff that relies on
userspace tools + qemu as well as just the kernel drivers.
Nevertheless a fast subset of libguestfs supporting just kernel
filesystem drivers could be useful.


LKL uses the full Linux I/O stack and I think LVM and partitioning
should work out of the box. Adding support for qcow2 and vmdk should
be possible as well. ntfs-3g might be problematic.


Partitioning will work fine based on what you say.  MD using the old 
metadata and automatic assembly should also work fine (assuming there is 
some way to tell the library to simulate running initcalls).  DM (and by 
extension LVM, which is a bunch of userspace stuff on top of regular DM) 
and new style MD both require userspace tools to configure and interact 
with, this could be handled in two different ways:

1. Update the LVM and MD tools so they can be built as libraries and
   work directly with LKL.
2. Provide some wrapper functions to emulate dmsetup, lvm, and mdadm as
   distinct library calls.
Of these, the first option is likely to be the best for long-term 
support, but the second is probably going to be easier to code quickly.


QCOW2 and VMDK are both VM disk formats, and while I would love to see a 
driver treat them (and VDI, and VHD) like regular disk images on Linux 
in general, that will take some effort to implement properly.


NTFS-3G is a FUSE based filesystem driver, so that kind of functionality 
would probably need to be implemented in the application itself 
(although having some way to have the app just link to it instead would 
be absolutely wonderful).





smime.p7s
Description: S/MIME Cryptographic Signature

Re: [RFC PATCH 00/28] Linux Kernel Library

[Octavian Purdila]

On Wed, Nov 4, 2015 at 3:50 PM, Richard W.M. Jones  wrote:
> On Wed, Nov 04, 2015 at 01:24:03AM +0200, Octavian Purdila wrote:
>> Thanks for the pointers Richard, I am going to take a look at it.
>
> Now I've had a chance to look at some of the example LKL tools, here's
> what this actually involves.  It's not actually a great deal of work,
> it could probably be done in a day or two, but see my question about
> `lkl_sys_*' below.
>
> libguestfs (the library part) needs to talk over an RPC connection to
> its daemon.  See diagram here:
>
> http://libguestfs.org/guestfs-internals.1.html
>
> The code in src/launch-{direct,libvirt,uml,...}.c sets up that
> connection and runs the daemon -- normally inside a qemu wrapper, but
> it could be inside UML.  For LKL I think it should just fork the
> daemon directly.
>
> The daemon would then be linked to LKL.
>
> So really what's needed is a src/launch-lkl.c probably modelled after
> one of these current backends:
>
> https://github.com/libguestfs/libguestfs/blob/master/src/launch-uml.c
> https://github.com/libguestfs/libguestfs/blob/master/src/launch-unix.c
>
> and then recompile the daemon to link to LKL:
>
> https://github.com/libguestfs/libguestfs/tree/master/daemon
>
> and pass the list of disk images to the daemon, probably best to do
> that on the guestfsd command line.
>
> My only problem here: you can't just link to daemon to LKL, do you
> have to change all of the system calls from `foo' to `lkl_sys_foo'?
> That's an awful lot of #ifdefs ...
>

We could redefine the syscalls/libc symbols to call lkl_sys_ functions
in launch-lkl, e.g.:

int opendir(const char *path)
{
   return lkl_opendir(new_path)
}
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Austin S Hemmelgarn]

On 2015-11-03 18:24, Octavian Purdila wrote:

On Wed, Nov 4, 2015 at 12:45 AM, Richard W.M. Jones  wrote:

I'm dubious that a lib-based approach could support LVM, partioning,
ntfs-3g, qcow2, vmdk and all the other libguestfs stuff that relies on
userspace tools + qemu as well as just the kernel drivers.
Nevertheless a fast subset of libguestfs supporting just kernel
filesystem drivers could be useful.


LKL uses the full Linux I/O stack and I think LVM and partitioning
should work out of the box. Adding support for qcow2 and vmdk should
be possible as well. ntfs-3g might be problematic.


Partitioning will work fine based on what you say.  MD using the old 
metadata and automatic assembly should also work fine (assuming there is 
some way to tell the library to simulate running initcalls).  DM (and by 
extension LVM, which is a bunch of userspace stuff on top of regular DM) 
and new style MD both require userspace tools to configure and interact 
with, this could be handled in two different ways:

1. Update the LVM and MD tools so they can be built as libraries and
   work directly with LKL.
2. Provide some wrapper functions to emulate dmsetup, lvm, and mdadm as
   distinct library calls.
Of these, the first option is likely to be the best for long-term 
support, but the second is probably going to be easier to code quickly.


QCOW2 and VMDK are both VM disk formats, and while I would love to see a 
driver treat them (and VDI, and VHD) like regular disk images on Linux 
in general, that will take some effort to implement properly.


NTFS-3G is a FUSE based filesystem driver, so that kind of functionality 
would probably need to be implemented in the application itself 
(although having some way to have the app just link to it instead would 
be absolutely wonderful).





smime.p7s
Description: S/MIME Cryptographic Signature

Re: [RFC PATCH 00/28] Linux Kernel Library

[Richard W.M. Jones]

On Wed, Nov 04, 2015 at 01:24:03AM +0200, Octavian Purdila wrote:
> Thanks for the pointers Richard, I am going to take a look at it.

Now I've had a chance to look at some of the example LKL tools, here's
what this actually involves.  It's not actually a great deal of work,
it could probably be done in a day or two, but see my question about
`lkl_sys_*' below.

libguestfs (the library part) needs to talk over an RPC connection to
its daemon.  See diagram here:

http://libguestfs.org/guestfs-internals.1.html

The code in src/launch-{direct,libvirt,uml,...}.c sets up that
connection and runs the daemon -- normally inside a qemu wrapper, but
it could be inside UML.  For LKL I think it should just fork the
daemon directly.

The daemon would then be linked to LKL.

So really what's needed is a src/launch-lkl.c probably modelled after
one of these current backends:

https://github.com/libguestfs/libguestfs/blob/master/src/launch-uml.c
https://github.com/libguestfs/libguestfs/blob/master/src/launch-unix.c

and then recompile the daemon to link to LKL:

https://github.com/libguestfs/libguestfs/tree/master/daemon

and pass the list of disk images to the daemon, probably best to do
that on the guestfsd command line.

My only problem here: you can't just link to daemon to LKL, do you
have to change all of the system calls from `foo' to `lkl_sys_foo'?
That's an awful lot of #ifdefs ...

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Octavian Purdila]

On Wed, Nov 4, 2015 at 12:45 AM, Richard W.M. Jones  wrote:
> On Tue, Nov 03, 2015 at 10:40:29PM +0100, Richard Weinberger wrote:
>> On Tue, Nov 3, 2015 at 9:20 PM, Octavian Purdila
>>  wrote:
>> > LKL (Linux Kernel Library) is aiming to allow reusing the Linux kernel code
>> > as extensively as possible with minimal effort and reduced maintenance
>> > overhead.
>> >
>> > Examples of how LKL can be used are: creating userspace applications
>> > (running on Linux and other operating systems) that can read or write Linux
>> > filesystems or can use the Linux networking stack, creating kernel drivers
>> > for other operating systems that can read Linux filesystems, bootloaders
>> > support for reading/writing Linux filesystems, etc.
>> >
>> > With LKL, the kernel code is compiled into an object file that can be
>> > directly linked by applications. The API offered by LKL is based on the
>> > Linux system call interface.
>> >
>> > LKL is implemented as an architecture port in arch/lkl. It relies on host
>> > operations defined by the application or a host library (tools/lkl/lib).
>> >
>> > The latest LKL version can be found at g...@github.com:lkl/linux.git
>>
>> Or more copy friendly: https://github.com/lkl/linux.git
>>
>> > FAQ
>> > ===
>> >
>> > Q: How is LKL different from UML?
>> > A: UML provides a full OS environment (e.g. user/kernel separation, user
>> > processes) and also has requirements (a filesystem, processes, etc.) that
>> > makes it hard to use it for standalone applications. UML also relies
>> > heavily on Linux hosts. On the other hand LKL is designed to be linked
>> > directly with the application and hence does not have user/kernel
>> > separation which makes it easier to use it in standalone applications.
>>
>> So, this is a "liblinux" where applications are directly linked
>> against the kernel.
>> IOW system calls are plain function calls into the kernel?
>>
>> This eliminates UML's most problematic areas, system call handling via 
>> ptrace()
>> and virtual memory management via SIGSEGV. :-)
>>
>> > Q: How is LKL different from LibOS?
>> > A: LibOS re-implements high-level kernel APIs for timers, softirqs,
>> > scheduling, sysctl, SLAB/SLUB, etc. LKL behaves like any arch port,
>> > implementing the arch level operations requested by the Linux kernel. LKL
>> > also offers a host interface so that support for multiple hosts can be
>> > easily implemented.
>>
>> Yeah, these re-implementations are what I find most worrisome about LibOS.
>>
>> >
>> > Building LKL the host library and LKL applications
>> > ==
>> >
>> > % cd tools/lkl
>> > % make
>> >
>> > will build LKL as a object file, it will install it in tools/lkl/lib 
>> > together
>> > with the headers files in tools/lkl/include then will build the host 
>> > library,
>> > tests and a few of application examples:
>> >
>> > * tests/boot - a simple applications that uses LKL and exercises the basic
>> > LKL APIs
>> >
>> > * fs2tar - a tool that converts a filesystem image to a tar archive
>> >
>> > * cptofs/cpfromfs - a tool that copies files to/from a filesystem image
>>
>> Seeing forward to have a libguestfs port. :-)
>
> Thanks - I was keeping an eye on libos (and on the NetBSD rump kernel
> stuff before), ready to integrate them into libguestfs as soon as they
> offered filesystem access.
>
> It's easy to write a libguestfs-compatible backend, which brings all
> the virt-* tools from libguestfs to the new code.  The UML one looks
> like this:
>
> https://github.com/libguestfs/libguestfs/blob/master/src/launch-uml.c
>

Thanks for the pointers Richard, I am going to take a look at it.

>
> I'm dubious that a lib-based approach could support LVM, partioning,
> ntfs-3g, qcow2, vmdk and all the other libguestfs stuff that relies on
> userspace tools + qemu as well as just the kernel drivers.
> Nevertheless a fast subset of libguestfs supporting just kernel
> filesystem drivers could be useful.
>

LKL uses the full Linux I/O stack and I think LVM and partitioning
should work out of the box. Adding support for qcow2 and vmdk should
be possible as well. ntfs-3g might be problematic.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Hajime Tazaki]

At Tue, 3 Nov 2015 22:45:45 +,
Richard W.M. Jones wrote:

> > > * cptofs/cpfromfs - a tool that copies files to/from a filesystem image
> > 
> > Seeing forward to have a libguestfs port. :-)
> 
> Thanks - I was keeping an eye on libos (and on the NetBSD rump kernel
> stuff before), ready to integrate them into libguestfs as soon as they
> offered filesystem access.

I've been working on fs support on libos recently during
the integration with rump kernel _hypercall_, though it's
still in the middle (but open(2) in a specific condition
works fine at least).

https://github.com/libos-nuse/net-next-nuse/tree/rump-hypcall

I would expect to see more concrete patchset in near future.

-- Hajime
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Octavian Purdila]

On Tue, Nov 3, 2015 at 11:40 PM, Richard Weinberger
 wrote:

Hi Richard,

> On Tue, Nov 3, 2015 at 9:20 PM, Octavian Purdila
>  wrote:
>> LKL (Linux Kernel Library) is aiming to allow reusing the Linux kernel code
>> as extensively as possible with minimal effort and reduced maintenance
>> overhead.
>>
>> Examples of how LKL can be used are: creating userspace applications
>> (running on Linux and other operating systems) that can read or write Linux
>> filesystems or can use the Linux networking stack, creating kernel drivers
>> for other operating systems that can read Linux filesystems, bootloaders
>> support for reading/writing Linux filesystems, etc.
>>
>> With LKL, the kernel code is compiled into an object file that can be
>> directly linked by applications. The API offered by LKL is based on the
>> Linux system call interface.
>>
>> LKL is implemented as an architecture port in arch/lkl. It relies on host
>> operations defined by the application or a host library (tools/lkl/lib).
>>
>> The latest LKL version can be found at g...@github.com:lkl/linux.git
>
> Or more copy friendly: https://github.com/lkl/linux.git
>
>> FAQ
>> ===
>>
>> Q: How is LKL different from UML?
>> A: UML provides a full OS environment (e.g. user/kernel separation, user
>> processes) and also has requirements (a filesystem, processes, etc.) that
>> makes it hard to use it for standalone applications. UML also relies
>> heavily on Linux hosts. On the other hand LKL is designed to be linked
>> directly with the application and hence does not have user/kernel
>> separation which makes it easier to use it in standalone applications.
>
> So, this is a "liblinux" where applications are directly linked
> against the kernel.
> IOW system calls are plain function calls into the kernel?
>

More like "thread" calls. All system calls are executed in a dedicate
(kernel) thread to avoid race conditions with the "interrupt" path.

> This eliminates UML's most problematic areas, system call handling via 
> ptrace()
> and virtual memory management via SIGSEGV. :-)
>

:)

>> Q: How is LKL different from LibOS?
>> A: LibOS re-implements high-level kernel APIs for timers, softirqs,
>> scheduling, sysctl, SLAB/SLUB, etc. LKL behaves like any arch port,
>> implementing the arch level operations requested by the Linux kernel. LKL
>> also offers a host interface so that support for multiple hosts can be
>> easily implemented.
>
> Yeah, these re-implementations are what I find most worrisome about LibOS.
>
>>
>> Building LKL the host library and LKL applications
>> ==
>>
>> % cd tools/lkl
>> % make
>>
>> will build LKL as a object file, it will install it in tools/lkl/lib together
>> with the headers files in tools/lkl/include then will build the host library,
>> tests and a few of application examples:
>>
>> * tests/boot - a simple applications that uses LKL and exercises the basic
>> LKL APIs
>>
>> * fs2tar - a tool that converts a filesystem image to a tar archive
>>
>> * cptofs/cpfromfs - a tool that copies files to/from a filesystem image
>
> Seeing forward to have a libguestfs port. :-)
>
> Is LKL strictly single threaded?
>

At this point yes. SMP support is on my todo list though :)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Richard W.M. Jones]

On Tue, Nov 03, 2015 at 10:40:29PM +0100, Richard Weinberger wrote:
> On Tue, Nov 3, 2015 at 9:20 PM, Octavian Purdila
>  wrote:
> > LKL (Linux Kernel Library) is aiming to allow reusing the Linux kernel code
> > as extensively as possible with minimal effort and reduced maintenance
> > overhead.
> >
> > Examples of how LKL can be used are: creating userspace applications
> > (running on Linux and other operating systems) that can read or write Linux
> > filesystems or can use the Linux networking stack, creating kernel drivers
> > for other operating systems that can read Linux filesystems, bootloaders
> > support for reading/writing Linux filesystems, etc.
> >
> > With LKL, the kernel code is compiled into an object file that can be
> > directly linked by applications. The API offered by LKL is based on the
> > Linux system call interface.
> >
> > LKL is implemented as an architecture port in arch/lkl. It relies on host
> > operations defined by the application or a host library (tools/lkl/lib).
> >
> > The latest LKL version can be found at g...@github.com:lkl/linux.git
> 
> Or more copy friendly: https://github.com/lkl/linux.git
> 
> > FAQ
> > ===
> >
> > Q: How is LKL different from UML?
> > A: UML provides a full OS environment (e.g. user/kernel separation, user
> > processes) and also has requirements (a filesystem, processes, etc.) that
> > makes it hard to use it for standalone applications. UML also relies
> > heavily on Linux hosts. On the other hand LKL is designed to be linked
> > directly with the application and hence does not have user/kernel
> > separation which makes it easier to use it in standalone applications.
> 
> So, this is a "liblinux" where applications are directly linked
> against the kernel.
> IOW system calls are plain function calls into the kernel?
> 
> This eliminates UML's most problematic areas, system call handling via 
> ptrace()
> and virtual memory management via SIGSEGV. :-)
> 
> > Q: How is LKL different from LibOS?
> > A: LibOS re-implements high-level kernel APIs for timers, softirqs,
> > scheduling, sysctl, SLAB/SLUB, etc. LKL behaves like any arch port,
> > implementing the arch level operations requested by the Linux kernel. LKL
> > also offers a host interface so that support for multiple hosts can be
> > easily implemented.
> 
> Yeah, these re-implementations are what I find most worrisome about LibOS.
> 
> >
> > Building LKL the host library and LKL applications
> > ==
> >
> > % cd tools/lkl
> > % make
> >
> > will build LKL as a object file, it will install it in tools/lkl/lib 
> > together
> > with the headers files in tools/lkl/include then will build the host 
> > library,
> > tests and a few of application examples:
> >
> > * tests/boot - a simple applications that uses LKL and exercises the basic
> > LKL APIs
> >
> > * fs2tar - a tool that converts a filesystem image to a tar archive
> >
> > * cptofs/cpfromfs - a tool that copies files to/from a filesystem image
> 
> Seeing forward to have a libguestfs port. :-)

Thanks - I was keeping an eye on libos (and on the NetBSD rump kernel
stuff before), ready to integrate them into libguestfs as soon as they
offered filesystem access.

It's easy to write a libguestfs-compatible backend, which brings all
the virt-* tools from libguestfs to the new code.  The UML one looks
like this:

https://github.com/libguestfs/libguestfs/blob/master/src/launch-uml.c

I'm dubious that a lib-based approach could support LVM, partioning,
ntfs-3g, qcow2, vmdk and all the other libguestfs stuff that relies on
userspace tools + qemu as well as just the kernel drivers.
Nevertheless a fast subset of libguestfs supporting just kernel
filesystem drivers could be useful.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Richard Weinberger]

On Tue, Nov 3, 2015 at 9:20 PM, Octavian Purdila
 wrote:
> LKL (Linux Kernel Library) is aiming to allow reusing the Linux kernel code
> as extensively as possible with minimal effort and reduced maintenance
> overhead.
>
> Examples of how LKL can be used are: creating userspace applications
> (running on Linux and other operating systems) that can read or write Linux
> filesystems or can use the Linux networking stack, creating kernel drivers
> for other operating systems that can read Linux filesystems, bootloaders
> support for reading/writing Linux filesystems, etc.
>
> With LKL, the kernel code is compiled into an object file that can be
> directly linked by applications. The API offered by LKL is based on the
> Linux system call interface.
>
> LKL is implemented as an architecture port in arch/lkl. It relies on host
> operations defined by the application or a host library (tools/lkl/lib).
>
> The latest LKL version can be found at g...@github.com:lkl/linux.git

Or more copy friendly: https://github.com/lkl/linux.git

> FAQ
> ===
>
> Q: How is LKL different from UML?
> A: UML provides a full OS environment (e.g. user/kernel separation, user
> processes) and also has requirements (a filesystem, processes, etc.) that
> makes it hard to use it for standalone applications. UML also relies
> heavily on Linux hosts. On the other hand LKL is designed to be linked
> directly with the application and hence does not have user/kernel
> separation which makes it easier to use it in standalone applications.

So, this is a "liblinux" where applications are directly linked
against the kernel.
IOW system calls are plain function calls into the kernel?

This eliminates UML's most problematic areas, system call handling via ptrace()
and virtual memory management via SIGSEGV. :-)

> Q: How is LKL different from LibOS?
> A: LibOS re-implements high-level kernel APIs for timers, softirqs,
> scheduling, sysctl, SLAB/SLUB, etc. LKL behaves like any arch port,
> implementing the arch level operations requested by the Linux kernel. LKL
> also offers a host interface so that support for multiple hosts can be
> easily implemented.

Yeah, these re-implementations are what I find most worrisome about LibOS.

>
> Building LKL the host library and LKL applications
> ==
>
> % cd tools/lkl
> % make
>
> will build LKL as a object file, it will install it in tools/lkl/lib together
> with the headers files in tools/lkl/include then will build the host library,
> tests and a few of application examples:
>
> * tests/boot - a simple applications that uses LKL and exercises the basic
> LKL APIs
>
> * fs2tar - a tool that converts a filesystem image to a tar archive
>
> * cptofs/cpfromfs - a tool that copies files to/from a filesystem image

Seeing forward to have a libguestfs port. :-)

Is LKL strictly single threaded?

-- 
Thanks,
//richard
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[RFC PATCH 00/28] Linux Kernel Library

[Octavian Purdila]

LKL (Linux Kernel Library) is aiming to allow reusing the Linux kernel code
as extensively as possible with minimal effort and reduced maintenance
overhead.

Examples of how LKL can be used are: creating userspace applications
(running on Linux and other operating systems) that can read or write Linux
filesystems or can use the Linux networking stack, creating kernel drivers
for other operating systems that can read Linux filesystems, bootloaders
support for reading/writing Linux filesystems, etc.

With LKL, the kernel code is compiled into an object file that can be
directly linked by applications. The API offered by LKL is based on the
Linux system call interface.

LKL is implemented as an architecture port in arch/lkl. It relies on host
operations defined by the application or a host library (tools/lkl/lib).

The latest LKL version can be found at g...@github.com:lkl/linux.git

FAQ
===

Q: How is LKL different from UML?
A: UML provides a full OS environment (e.g. user/kernel separation, user
processes) and also has requirements (a filesystem, processes, etc.) that
makes it hard to use it for standalone applications. UML also relies
heavily on Linux hosts. On the other hand LKL is designed to be linked
directly with the application and hence does not have user/kernel
separation which makes it easier to use it in standalone applications.

Q: How is LKL different from LibOS?
A: LibOS re-implements high-level kernel APIs for timers, softirqs,
scheduling, sysctl, SLAB/SLUB, etc. LKL behaves like any arch port,
implementing the arch level operations requested by the Linux kernel. LKL
also offers a host interface so that support for multiple hosts can be
easily implemented.


Building LKL the host library and LKL applications
==

% cd tools/lkl
% make

will build LKL as a object file, it will install it in tools/lkl/lib together
with the headers files in tools/lkl/include then will build the host library,
tests and a few of application examples:

* tests/boot - a simple applications that uses LKL and exercises the basic
LKL APIs

* fs2tar - a tool that converts a filesystem image to a tar archive

* cptofs/cpfromfs - a tool that copies files to/from a filesystem image


Supported hosts
===

Currently LKL supports POSIX and Windows userspace applications. New hosts
can be added relatively easy if the host supports gcc and GNU ld. Previous
versions of LKL supported Windows kernel and Haiku kernel hosts.


Octavian Purdila (28):
  asm-generic: atomic64: allow using generic atomic64 on 64bit platforms
  kbuild: allow architectures to automatically define kconfig symbols
  lkl: architecture skeleton for Linux kernel library
  lkl: host interface
  lkl: memory handling
  lkl: kernel threads support
  lkl: interrupt support
  lkl: system call interface and application API
  lkl: timers, time and delay support
  lkl: memory mapped I/O support
  lkl: basic kernel console support
  init: allow architecture code to overide run_init_process
  lkl: initialization and cleanup
  lkl: plug in the build system
  lkl tools: skeleton for host side library, tests and tools
  lkl tools: host lib: add lkl_strerror and lkl_printf
  lkl tools: host lib: memory mapped I/O helpers
  lkl tools: host lib: virtio devices
  lkl tools: host lib: virtio block device
  lkl tools: host lib: filesystem helpers
  lkl tools: host lib: posix host operations
  lkl tools: "boot" test
  lkl tools: tool that converts a filesystem image to tar
  lkl tools: tool that reads/writes to/from a filesystem image
  signal: use CONFIG_X86_32 instead of __i386__
  asm-generic: vmlinux.lds.h: allow customized rodata section name
  lkl: add support for Windows hosts
  lkl tools: add support for Windows host

 MAINTAINERS |   6 +
 Makefile|   1 +
 arch/lkl/.gitignore |   1 +
 arch/lkl/Kconfig|  83 ++
 arch/lkl/Makefile   |  39 +++
 arch/lkl/auto.conf  |   1 +
 arch/lkl/defconfig  |  35 +++
 arch/lkl/include/asm/Kbuild |  77 +
 arch/lkl/include/asm/bitsperlong.h  |  11 +
 arch/lkl/include/asm/byteorder.h|  10 +
 arch/lkl/include/asm/dma-mapping.h  |   6 +
 arch/lkl/include/asm/elf.h  |  13 +
 arch/lkl/include/asm/host_ops.h |   9 +
 arch/lkl/include/asm/io.h   | 104 +++
 arch/lkl/include/asm/irq.h  |  10 +
 arch/lkl/include/asm/mutex.h|   7 +
 arch/lkl/include/asm/page.h |  13 +
 arch/lkl/include/asm/pgtable.h  |  60 
 arch/lkl/include/asm/processor.h|  53 
 arch/lkl/include/asm/ptrace.h   |  23 ++
 arch/lkl/include/asm/setup.h|  12 +
 arch/lkl/include/asm/thread_info.h  |  82 +
 arch/lkl/include/asm/unistd.h   |  93 ++
 arch/lkl/include/asm/vmlinux.lds.h  |  10 +
 

Re: [RFC PATCH 00/28] Linux Kernel Library

[Richard Weinberger]

On Tue, Nov 3, 2015 at 9:20 PM, Octavian Purdila
 wrote:
> LKL (Linux Kernel Library) is aiming to allow reusing the Linux kernel code
> as extensively as possible with minimal effort and reduced maintenance
> overhead.
>
> Examples of how LKL can be used are: creating userspace applications
> (running on Linux and other operating systems) that can read or write Linux
> filesystems or can use the Linux networking stack, creating kernel drivers
> for other operating systems that can read Linux filesystems, bootloaders
> support for reading/writing Linux filesystems, etc.
>
> With LKL, the kernel code is compiled into an object file that can be
> directly linked by applications. The API offered by LKL is based on the
> Linux system call interface.
>
> LKL is implemented as an architecture port in arch/lkl. It relies on host
> operations defined by the application or a host library (tools/lkl/lib).
>
> The latest LKL version can be found at g...@github.com:lkl/linux.git

Or more copy friendly: https://github.com/lkl/linux.git

> FAQ
> ===
>
> Q: How is LKL different from UML?
> A: UML provides a full OS environment (e.g. user/kernel separation, user
> processes) and also has requirements (a filesystem, processes, etc.) that
> makes it hard to use it for standalone applications. UML also relies
> heavily on Linux hosts. On the other hand LKL is designed to be linked
> directly with the application and hence does not have user/kernel
> separation which makes it easier to use it in standalone applications.

So, this is a "liblinux" where applications are directly linked
against the kernel.
IOW system calls are plain function calls into the kernel?

This eliminates UML's most problematic areas, system call handling via ptrace()
and virtual memory management via SIGSEGV. :-)

> Q: How is LKL different from LibOS?
> A: LibOS re-implements high-level kernel APIs for timers, softirqs,
> scheduling, sysctl, SLAB/SLUB, etc. LKL behaves like any arch port,
> implementing the arch level operations requested by the Linux kernel. LKL
> also offers a host interface so that support for multiple hosts can be
> easily implemented.

Yeah, these re-implementations are what I find most worrisome about LibOS.

>
> Building LKL the host library and LKL applications
> ==
>
> % cd tools/lkl
> % make
>
> will build LKL as a object file, it will install it in tools/lkl/lib together
> with the headers files in tools/lkl/include then will build the host library,
> tests and a few of application examples:
>
> * tests/boot - a simple applications that uses LKL and exercises the basic
> LKL APIs
>
> * fs2tar - a tool that converts a filesystem image to a tar archive
>
> * cptofs/cpfromfs - a tool that copies files to/from a filesystem image

Seeing forward to have a libguestfs port. :-)

Is LKL strictly single threaded?

-- 
Thanks,
//richard
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Octavian Purdila]

On Tue, Nov 3, 2015 at 11:40 PM, Richard Weinberger
 wrote:

Hi Richard,

> On Tue, Nov 3, 2015 at 9:20 PM, Octavian Purdila
>  wrote:
>> LKL (Linux Kernel Library) is aiming to allow reusing the Linux kernel code
>> as extensively as possible with minimal effort and reduced maintenance
>> overhead.
>>
>> Examples of how LKL can be used are: creating userspace applications
>> (running on Linux and other operating systems) that can read or write Linux
>> filesystems or can use the Linux networking stack, creating kernel drivers
>> for other operating systems that can read Linux filesystems, bootloaders
>> support for reading/writing Linux filesystems, etc.
>>
>> With LKL, the kernel code is compiled into an object file that can be
>> directly linked by applications. The API offered by LKL is based on the
>> Linux system call interface.
>>
>> LKL is implemented as an architecture port in arch/lkl. It relies on host
>> operations defined by the application or a host library (tools/lkl/lib).
>>
>> The latest LKL version can be found at g...@github.com:lkl/linux.git
>
> Or more copy friendly: https://github.com/lkl/linux.git
>
>> FAQ
>> ===
>>
>> Q: How is LKL different from UML?
>> A: UML provides a full OS environment (e.g. user/kernel separation, user
>> processes) and also has requirements (a filesystem, processes, etc.) that
>> makes it hard to use it for standalone applications. UML also relies
>> heavily on Linux hosts. On the other hand LKL is designed to be linked
>> directly with the application and hence does not have user/kernel
>> separation which makes it easier to use it in standalone applications.
>
> So, this is a "liblinux" where applications are directly linked
> against the kernel.
> IOW system calls are plain function calls into the kernel?
>

More like "thread" calls. All system calls are executed in a dedicate
(kernel) thread to avoid race conditions with the "interrupt" path.

> This eliminates UML's most problematic areas, system call handling via 
> ptrace()
> and virtual memory management via SIGSEGV. :-)
>

:)

>> Q: How is LKL different from LibOS?
>> A: LibOS re-implements high-level kernel APIs for timers, softirqs,
>> scheduling, sysctl, SLAB/SLUB, etc. LKL behaves like any arch port,
>> implementing the arch level operations requested by the Linux kernel. LKL
>> also offers a host interface so that support for multiple hosts can be
>> easily implemented.
>
> Yeah, these re-implementations are what I find most worrisome about LibOS.
>
>>
>> Building LKL the host library and LKL applications
>> ==
>>
>> % cd tools/lkl
>> % make
>>
>> will build LKL as a object file, it will install it in tools/lkl/lib together
>> with the headers files in tools/lkl/include then will build the host library,
>> tests and a few of application examples:
>>
>> * tests/boot - a simple applications that uses LKL and exercises the basic
>> LKL APIs
>>
>> * fs2tar - a tool that converts a filesystem image to a tar archive
>>
>> * cptofs/cpfromfs - a tool that copies files to/from a filesystem image
>
> Seeing forward to have a libguestfs port. :-)
>
> Is LKL strictly single threaded?
>

At this point yes. SMP support is on my todo list though :)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Richard W.M. Jones]

On Tue, Nov 03, 2015 at 10:40:29PM +0100, Richard Weinberger wrote:
> On Tue, Nov 3, 2015 at 9:20 PM, Octavian Purdila
>  wrote:
> > LKL (Linux Kernel Library) is aiming to allow reusing the Linux kernel code
> > as extensively as possible with minimal effort and reduced maintenance
> > overhead.
> >
> > Examples of how LKL can be used are: creating userspace applications
> > (running on Linux and other operating systems) that can read or write Linux
> > filesystems or can use the Linux networking stack, creating kernel drivers
> > for other operating systems that can read Linux filesystems, bootloaders
> > support for reading/writing Linux filesystems, etc.
> >
> > With LKL, the kernel code is compiled into an object file that can be
> > directly linked by applications. The API offered by LKL is based on the
> > Linux system call interface.
> >
> > LKL is implemented as an architecture port in arch/lkl. It relies on host
> > operations defined by the application or a host library (tools/lkl/lib).
> >
> > The latest LKL version can be found at g...@github.com:lkl/linux.git
> 
> Or more copy friendly: https://github.com/lkl/linux.git
> 
> > FAQ
> > ===
> >
> > Q: How is LKL different from UML?
> > A: UML provides a full OS environment (e.g. user/kernel separation, user
> > processes) and also has requirements (a filesystem, processes, etc.) that
> > makes it hard to use it for standalone applications. UML also relies
> > heavily on Linux hosts. On the other hand LKL is designed to be linked
> > directly with the application and hence does not have user/kernel
> > separation which makes it easier to use it in standalone applications.
> 
> So, this is a "liblinux" where applications are directly linked
> against the kernel.
> IOW system calls are plain function calls into the kernel?
> 
> This eliminates UML's most problematic areas, system call handling via 
> ptrace()
> and virtual memory management via SIGSEGV. :-)
> 
> > Q: How is LKL different from LibOS?
> > A: LibOS re-implements high-level kernel APIs for timers, softirqs,
> > scheduling, sysctl, SLAB/SLUB, etc. LKL behaves like any arch port,
> > implementing the arch level operations requested by the Linux kernel. LKL
> > also offers a host interface so that support for multiple hosts can be
> > easily implemented.
> 
> Yeah, these re-implementations are what I find most worrisome about LibOS.
> 
> >
> > Building LKL the host library and LKL applications
> > ==
> >
> > % cd tools/lkl
> > % make
> >
> > will build LKL as a object file, it will install it in tools/lkl/lib 
> > together
> > with the headers files in tools/lkl/include then will build the host 
> > library,
> > tests and a few of application examples:
> >
> > * tests/boot - a simple applications that uses LKL and exercises the basic
> > LKL APIs
> >
> > * fs2tar - a tool that converts a filesystem image to a tar archive
> >
> > * cptofs/cpfromfs - a tool that copies files to/from a filesystem image
> 
> Seeing forward to have a libguestfs port. :-)

Thanks - I was keeping an eye on libos (and on the NetBSD rump kernel
stuff before), ready to integrate them into libguestfs as soon as they
offered filesystem access.

It's easy to write a libguestfs-compatible backend, which brings all
the virt-* tools from libguestfs to the new code.  The UML one looks
like this:

https://github.com/libguestfs/libguestfs/blob/master/src/launch-uml.c

I'm dubious that a lib-based approach could support LVM, partioning,
ntfs-3g, qcow2, vmdk and all the other libguestfs stuff that relies on
userspace tools + qemu as well as just the kernel drivers.
Nevertheless a fast subset of libguestfs supporting just kernel
filesystem drivers could be useful.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[RFC PATCH 00/28] Linux Kernel Library

[Octavian Purdila]

LKL (Linux Kernel Library) is aiming to allow reusing the Linux kernel code
as extensively as possible with minimal effort and reduced maintenance
overhead.

Examples of how LKL can be used are: creating userspace applications
(running on Linux and other operating systems) that can read or write Linux
filesystems or can use the Linux networking stack, creating kernel drivers
for other operating systems that can read Linux filesystems, bootloaders
support for reading/writing Linux filesystems, etc.

With LKL, the kernel code is compiled into an object file that can be
directly linked by applications. The API offered by LKL is based on the
Linux system call interface.

LKL is implemented as an architecture port in arch/lkl. It relies on host
operations defined by the application or a host library (tools/lkl/lib).

The latest LKL version can be found at g...@github.com:lkl/linux.git

FAQ
===

Q: How is LKL different from UML?
A: UML provides a full OS environment (e.g. user/kernel separation, user
processes) and also has requirements (a filesystem, processes, etc.) that
makes it hard to use it for standalone applications. UML also relies
heavily on Linux hosts. On the other hand LKL is designed to be linked
directly with the application and hence does not have user/kernel
separation which makes it easier to use it in standalone applications.

Q: How is LKL different from LibOS?
A: LibOS re-implements high-level kernel APIs for timers, softirqs,
scheduling, sysctl, SLAB/SLUB, etc. LKL behaves like any arch port,
implementing the arch level operations requested by the Linux kernel. LKL
also offers a host interface so that support for multiple hosts can be
easily implemented.


Building LKL the host library and LKL applications
==

% cd tools/lkl
% make

will build LKL as a object file, it will install it in tools/lkl/lib together
with the headers files in tools/lkl/include then will build the host library,
tests and a few of application examples:

* tests/boot - a simple applications that uses LKL and exercises the basic
LKL APIs

* fs2tar - a tool that converts a filesystem image to a tar archive

* cptofs/cpfromfs - a tool that copies files to/from a filesystem image


Supported hosts
===

Currently LKL supports POSIX and Windows userspace applications. New hosts
can be added relatively easy if the host supports gcc and GNU ld. Previous
versions of LKL supported Windows kernel and Haiku kernel hosts.


Octavian Purdila (28):
  asm-generic: atomic64: allow using generic atomic64 on 64bit platforms
  kbuild: allow architectures to automatically define kconfig symbols
  lkl: architecture skeleton for Linux kernel library
  lkl: host interface
  lkl: memory handling
  lkl: kernel threads support
  lkl: interrupt support
  lkl: system call interface and application API
  lkl: timers, time and delay support
  lkl: memory mapped I/O support
  lkl: basic kernel console support
  init: allow architecture code to overide run_init_process
  lkl: initialization and cleanup
  lkl: plug in the build system
  lkl tools: skeleton for host side library, tests and tools
  lkl tools: host lib: add lkl_strerror and lkl_printf
  lkl tools: host lib: memory mapped I/O helpers
  lkl tools: host lib: virtio devices
  lkl tools: host lib: virtio block device
  lkl tools: host lib: filesystem helpers
  lkl tools: host lib: posix host operations
  lkl tools: "boot" test
  lkl tools: tool that converts a filesystem image to tar
  lkl tools: tool that reads/writes to/from a filesystem image
  signal: use CONFIG_X86_32 instead of __i386__
  asm-generic: vmlinux.lds.h: allow customized rodata section name
  lkl: add support for Windows hosts
  lkl tools: add support for Windows host

 MAINTAINERS |   6 +
 Makefile|   1 +
 arch/lkl/.gitignore |   1 +
 arch/lkl/Kconfig|  83 ++
 arch/lkl/Makefile   |  39 +++
 arch/lkl/auto.conf  |   1 +
 arch/lkl/defconfig  |  35 +++
 arch/lkl/include/asm/Kbuild |  77 +
 arch/lkl/include/asm/bitsperlong.h  |  11 +
 arch/lkl/include/asm/byteorder.h|  10 +
 arch/lkl/include/asm/dma-mapping.h  |   6 +
 arch/lkl/include/asm/elf.h  |  13 +
 arch/lkl/include/asm/host_ops.h |   9 +
 arch/lkl/include/asm/io.h   | 104 +++
 arch/lkl/include/asm/irq.h  |  10 +
 arch/lkl/include/asm/mutex.h|   7 +
 arch/lkl/include/asm/page.h |  13 +
 arch/lkl/include/asm/pgtable.h  |  60 
 arch/lkl/include/asm/processor.h|  53 
 arch/lkl/include/asm/ptrace.h   |  23 ++
 arch/lkl/include/asm/setup.h|  12 +
 arch/lkl/include/asm/thread_info.h  |  82 +
 arch/lkl/include/asm/unistd.h   |  93 ++
 arch/lkl/include/asm/vmlinux.lds.h  |  10 +
 

Re: [RFC PATCH 00/28] Linux Kernel Library

[Octavian Purdila]

On Wed, Nov 4, 2015 at 12:45 AM, Richard W.M. Jones  wrote:
> On Tue, Nov 03, 2015 at 10:40:29PM +0100, Richard Weinberger wrote:
>> On Tue, Nov 3, 2015 at 9:20 PM, Octavian Purdila
>>  wrote:
>> > LKL (Linux Kernel Library) is aiming to allow reusing the Linux kernel code
>> > as extensively as possible with minimal effort and reduced maintenance
>> > overhead.
>> >
>> > Examples of how LKL can be used are: creating userspace applications
>> > (running on Linux and other operating systems) that can read or write Linux
>> > filesystems or can use the Linux networking stack, creating kernel drivers
>> > for other operating systems that can read Linux filesystems, bootloaders
>> > support for reading/writing Linux filesystems, etc.
>> >
>> > With LKL, the kernel code is compiled into an object file that can be
>> > directly linked by applications. The API offered by LKL is based on the
>> > Linux system call interface.
>> >
>> > LKL is implemented as an architecture port in arch/lkl. It relies on host
>> > operations defined by the application or a host library (tools/lkl/lib).
>> >
>> > The latest LKL version can be found at g...@github.com:lkl/linux.git
>>
>> Or more copy friendly: https://github.com/lkl/linux.git
>>
>> > FAQ
>> > ===
>> >
>> > Q: How is LKL different from UML?
>> > A: UML provides a full OS environment (e.g. user/kernel separation, user
>> > processes) and also has requirements (a filesystem, processes, etc.) that
>> > makes it hard to use it for standalone applications. UML also relies
>> > heavily on Linux hosts. On the other hand LKL is designed to be linked
>> > directly with the application and hence does not have user/kernel
>> > separation which makes it easier to use it in standalone applications.
>>
>> So, this is a "liblinux" where applications are directly linked
>> against the kernel.
>> IOW system calls are plain function calls into the kernel?
>>
>> This eliminates UML's most problematic areas, system call handling via 
>> ptrace()
>> and virtual memory management via SIGSEGV. :-)
>>
>> > Q: How is LKL different from LibOS?
>> > A: LibOS re-implements high-level kernel APIs for timers, softirqs,
>> > scheduling, sysctl, SLAB/SLUB, etc. LKL behaves like any arch port,
>> > implementing the arch level operations requested by the Linux kernel. LKL
>> > also offers a host interface so that support for multiple hosts can be
>> > easily implemented.
>>
>> Yeah, these re-implementations are what I find most worrisome about LibOS.
>>
>> >
>> > Building LKL the host library and LKL applications
>> > ==
>> >
>> > % cd tools/lkl
>> > % make
>> >
>> > will build LKL as a object file, it will install it in tools/lkl/lib 
>> > together
>> > with the headers files in tools/lkl/include then will build the host 
>> > library,
>> > tests and a few of application examples:
>> >
>> > * tests/boot - a simple applications that uses LKL and exercises the basic
>> > LKL APIs
>> >
>> > * fs2tar - a tool that converts a filesystem image to a tar archive
>> >
>> > * cptofs/cpfromfs - a tool that copies files to/from a filesystem image
>>
>> Seeing forward to have a libguestfs port. :-)
>
> Thanks - I was keeping an eye on libos (and on the NetBSD rump kernel
> stuff before), ready to integrate them into libguestfs as soon as they
> offered filesystem access.
>
> It's easy to write a libguestfs-compatible backend, which brings all
> the virt-* tools from libguestfs to the new code.  The UML one looks
> like this:
>
> https://github.com/libguestfs/libguestfs/blob/master/src/launch-uml.c
>

Thanks for the pointers Richard, I am going to take a look at it.

>
> I'm dubious that a lib-based approach could support LVM, partioning,
> ntfs-3g, qcow2, vmdk and all the other libguestfs stuff that relies on
> userspace tools + qemu as well as just the kernel drivers.
> Nevertheless a fast subset of libguestfs supporting just kernel
> filesystem drivers could be useful.
>

LKL uses the full Linux I/O stack and I think LVM and partitioning
should work out of the box. Adding support for qcow2 and vmdk should
be possible as well. ntfs-3g might be problematic.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [RFC PATCH 00/28] Linux Kernel Library

[Hajime Tazaki]

At Tue, 3 Nov 2015 22:45:45 +,
Richard W.M. Jones wrote:

> > > * cptofs/cpfromfs - a tool that copies files to/from a filesystem image
> > 
> > Seeing forward to have a libguestfs port. :-)
> 
> Thanks - I was keeping an eye on libos (and on the NetBSD rump kernel
> stuff before), ready to integrate them into libguestfs as soon as they
> offered filesystem access.

I've been working on fs support on libos recently during
the integration with rump kernel _hypercall_, though it's
still in the middle (but open(2) in a specific condition
works fine at least).

https://github.com/libos-nuse/net-next-nuse/tree/rump-hypcall

I would expect to see more concrete patchset in near future.

-- Hajime
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/