Re: [GIT PULL] kdbus for 4.1-rc1

[Johannes Stezenbach]

On Mon, Apr 27, 2015 at 03:14:49PM -0700, Linus Torvalds wrote:
> On Mon, Apr 27, 2015 at 3:00 PM, Linus Torvalds
>  wrote:
> >
> > IOW, all the people who say that it's about avoiding context switches
> > are probably just full of shit. It's not about context switches, it's
> > about bad user-level code.
> 
> Just to make sure, I did a system-wide profile (so that you can
> actually see the overhead of context switching better), and that
> didn't change the picture.
> 
> The scheduler overhead *might* be 1% or so.
> 
> So really. The people who talk about how kdbus improves performance
> are just full of sh*t. Yes, it improves things, but the improvement
> seems to be 100% "incidental", in that it avoids a few trips down the
> user-space problems.

I was interested how plain UDS performs compared to the
dbus-client/dbus-server benchmark when doing a similar
transaction (RPC call from client1 to client2 via a server,
i.e 4 send() and 4 recv() syscalls per RPC msg).
Since I had worked on socket code for some project anyway, I
decided to write a stupid little benchmark.

On my machine, dbus-client/dbus-server needs ~200us per call (1024 byte msg),
UDS "dbus call" needs ~23us.  Of course, someone who cares about performance
wouldn't use sync RPC via a message broker, so I added
single-client and async mode to the benchmark for comparison.
Async mode not only decreases scheduling overhead, it also
can use two CPU cores, so it's more than twice as fast.

  ./server dbus
  (you need to run two clients, the timing loop starts
  when the second client connects)
  ./client sync 4096 100
 22.757250 s, 43942 msg/s, 22.8 us/msg, 171.638 MB/s
  ./client async 4096 100
 8.197482 s, 121989 msg/s, 8.2 us/msg, 476.488 MB/s
  ./server single
  (only a single client talks to the server)
  ./client sync 4096 100
 10.980143 s, 91073 msg/s, 11.0 us/msg, 355.733 MB/s
  ./client async 4096 100
 3.041953 s, 328736 msg/s, 3.0 us/msg, 1284.044 MB/s

In all cases 1 msg means "send request + receive response".


Johannes
/* UDS server */

#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 


// use abstract address
#define SOCKET "\0udsbench.socket"

static int die(const char *msg)
{
	if (errno)
		fprintf(stderr, "%s: error %d %m\n", msg, errno);
	else
		fprintf(stderr, "%s\n", msg);
	if (errno != ECONNRESET)
		exit(EXIT_FAILURE);
	return 0;
}

int main(int argc, char *argv[])
{
	struct sockaddr_un addr = {
		.sun_family = AF_UNIX,
		.sun_path = SOCKET,
	};
	int sock, client1, client2 = -1, rc, len;
	struct pollfd pfd[2];
	char buf[65536];
	unsigned long cnt = 0;
	bool single = false;

	if (argc != 2)
		die("usage: server {single|dbus}");
	if (!strcmp(argv[1], "single"))
		single = true;
	printf("running in %s mode\n", single ? "single" : "dbus");

	sock = socket(AF_UNIX, SOCK_SEQPACKET, 0);
	if (sock < 0)
		die("can't create socket");
	if (bind(sock, (struct sockaddr *) , sizeof(addr)) < 0)
		die("can't bind address");
	if (listen(sock, 5) < 0)
		die("can't listen");

	printf("waiting for client 1\n");
	client1 = accept(sock, NULL, NULL);
	if (client1 < 0)
		die("accept");

	if (!single) {
		printf("waiting for client 2\n");
		client2 = accept(sock, NULL, NULL);
		if (client2 < 0)
			die("accept");

		write(client2, "\01", 1);
	}
	write(client1, "\0", 1);

	printf("enter event loop\n");
	pfd[0].fd = client1;
	pfd[1].fd = client2;
	pfd[0].events = pfd[1].events = POLLIN;
	for (;;) {
		rc = poll(pfd, single ? 1 : 2, -1);
		if (rc < 0)
			die("poll");
		if (pfd[0].revents & POLLIN) {
			len = read(client1, buf, sizeof(buf));
			if (len < 0) {
die("read from client 1");
break;
			}
			if (len == 0) {
printf("client 1 EOF\n");
break;
			}
			rc = write(single ? client1 : client2, buf, len);
			if (len != rc) {
die("write to client 2");
break;
			}
			cnt++;
		}
		if (pfd[1].revents & POLLIN) {
			len = read(client2, buf, sizeof(buf));
			if (len < 0) {
die("read from client 2");
break;
			}
			if (len == 0) {
printf("client 2 EOF\n");
break;
			}
			rc = write(client1, buf, len);
			if (len != rc) {
die("write to client 1");
break;
			}
			cnt++;
		}
	}
	printf("passed %lu messages\n", cnt);
	return EXIT_SUCCESS;
}
/* UDS client */

#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 
#include 


// use abstract address
#define SOCKET "\0udsbench.socket"

static int die(const char *msg)
{
	if (errno)
		fprintf(stderr, "%s: error %d %m\n", msg, errno);
	else
		fprintf(stderr, "%s\n", msg);
	if (errno != EPIPE)
		exit(EXIT_FAILURE);
	return 0;
}

int main(int argc, char *argv[])
{
	struct sockaddr_un addr = {
		.sun_family = AF_UNIX,
		.sun_path = SOCKET,
	};
	int sock, rc, client = 1, i;
	char *buf;
	struct timespec start, end;
	double duration;
	bool async = false;
	struct pollfd pfd;
	typeof() f1, f2;
	long 

Re: [GIT PULL] kdbus for 4.1-rc1

[Johannes Stezenbach]

On Mon, Apr 27, 2015 at 03:14:49PM -0700, Linus Torvalds wrote:
 On Mon, Apr 27, 2015 at 3:00 PM, Linus Torvalds
 torva...@linux-foundation.org wrote:
 
  IOW, all the people who say that it's about avoiding context switches
  are probably just full of shit. It's not about context switches, it's
  about bad user-level code.
 
 Just to make sure, I did a system-wide profile (so that you can
 actually see the overhead of context switching better), and that
 didn't change the picture.
 
 The scheduler overhead *might* be 1% or so.
 
 So really. The people who talk about how kdbus improves performance
 are just full of sh*t. Yes, it improves things, but the improvement
 seems to be 100% incidental, in that it avoids a few trips down the
 user-space problems.

I was interested how plain UDS performs compared to the
dbus-client/dbus-server benchmark when doing a similar
transaction (RPC call from client1 to client2 via a server,
i.e 4 send() and 4 recv() syscalls per RPC msg).
Since I had worked on socket code for some project anyway, I
decided to write a stupid little benchmark.

On my machine, dbus-client/dbus-server needs ~200us per call (1024 byte msg),
UDS dbus call needs ~23us.  Of course, someone who cares about performance
wouldn't use sync RPC via a message broker, so I added
single-client and async mode to the benchmark for comparison.
Async mode not only decreases scheduling overhead, it also
can use two CPU cores, so it's more than twice as fast.

  ./server dbus
  (you need to run two clients, the timing loop starts
  when the second client connects)
  ./client sync 4096 100
 22.757250 s, 43942 msg/s, 22.8 us/msg, 171.638 MB/s
  ./client async 4096 100
 8.197482 s, 121989 msg/s, 8.2 us/msg, 476.488 MB/s
  ./server single
  (only a single client talks to the server)
  ./client sync 4096 100
 10.980143 s, 91073 msg/s, 11.0 us/msg, 355.733 MB/s
  ./client async 4096 100
 3.041953 s, 328736 msg/s, 3.0 us/msg, 1284.044 MB/s

In all cases 1 msg means send request + receive response.


Johannes
/* UDS server */

#include errno.h
#include poll.h
#include stdbool.h
#include stdio.h
#include stdlib.h
#include string.h
#include unistd.h
#include sys/socket.h
#include sys/timerfd.h
#include sys/un.h


// use abstract address
#define SOCKET \0udsbench.socket

static int die(const char *msg)
{
	if (errno)
		fprintf(stderr, %s: error %d %m\n, msg, errno);
	else
		fprintf(stderr, %s\n, msg);
	if (errno != ECONNRESET)
		exit(EXIT_FAILURE);
	return 0;
}

int main(int argc, char *argv[])
{
	struct sockaddr_un addr = {
		.sun_family = AF_UNIX,
		.sun_path = SOCKET,
	};
	int sock, client1, client2 = -1, rc, len;
	struct pollfd pfd[2];
	char buf[65536];
	unsigned long cnt = 0;
	bool single = false;

	if (argc != 2)
		die(usage: server {single|dbus});
	if (!strcmp(argv[1], single))
		single = true;
	printf(running in %s mode\n, single ? single : dbus);

	sock = socket(AF_UNIX, SOCK_SEQPACKET, 0);
	if (sock  0)
		die(can't create socket);
	if (bind(sock, (struct sockaddr *) addr, sizeof(addr))  0)
		die(can't bind address);
	if (listen(sock, 5)  0)
		die(can't listen);

	printf(waiting for client 1\n);
	client1 = accept(sock, NULL, NULL);
	if (client1  0)
		die(accept);

	if (!single) {
		printf(waiting for client 2\n);
		client2 = accept(sock, NULL, NULL);
		if (client2  0)
			die(accept);

		write(client2, \01, 1);
	}
	write(client1, \0, 1);

	printf(enter event loop\n);
	pfd[0].fd = client1;
	pfd[1].fd = client2;
	pfd[0].events = pfd[1].events = POLLIN;
	for (;;) {
		rc = poll(pfd, single ? 1 : 2, -1);
		if (rc  0)
			die(poll);
		if (pfd[0].revents  POLLIN) {
			len = read(client1, buf, sizeof(buf));
			if (len  0) {
die(read from client 1);
break;
			}
			if (len == 0) {
printf(client 1 EOF\n);
break;
			}
			rc = write(single ? client1 : client2, buf, len);
			if (len != rc) {
die(write to client 2);
break;
			}
			cnt++;
		}
		if (pfd[1].revents  POLLIN) {
			len = read(client2, buf, sizeof(buf));
			if (len  0) {
die(read from client 2);
break;
			}
			if (len == 0) {
printf(client 2 EOF\n);
break;
			}
			rc = write(client1, buf, len);
			if (len != rc) {
die(write to client 1);
break;
			}
			cnt++;
		}
	}
	printf(passed %lu messages\n, cnt);
	return EXIT_SUCCESS;
}
/* UDS client */

#include alloca.h
#include errno.h
#include poll.h
#include stdbool.h
#include stdio.h
#include stdlib.h
#include string.h
#include time.h
#include unistd.h
#include sys/socket.h
#include sys/timerfd.h
#include sys/un.h


// use abstract address
#define SOCKET \0udsbench.socket

static int die(const char *msg)
{
	if (errno)
		fprintf(stderr, %s: error %d %m\n, msg, errno);
	else
		fprintf(stderr, %s\n, msg);
	if (errno != EPIPE)
		exit(EXIT_FAILURE);
	return 0;
}

int main(int argc, char *argv[])
{
	struct sockaddr_un addr = {
		.sun_family = AF_UNIX,
		.sun_path = SOCKET,
	};
	int sock, rc, client = 1, i;
	

Re: [GIT PULL] kdbus for 4.1-rc1

[cee1]

2015-04-30 22:52 GMT+08:00 Łukasz Stelmach :
> It was <2015-04-30 czw 14:45>, when Richard Weinberger wrote:
>> Am 30.04.2015 um 14:40 schrieb Łukasz Stelmach:
>>> It was <2015-04-30 czw 14:23>, when Richard Weinberger wrote:
 Am 30.04.2015 um 14:16 schrieb Łukasz Stelmach:
> It was <2015-04-30 czw 12:40>, when Richard Weinberger wrote:
>> Am 30.04.2015 um 12:19 schrieb Łukasz Stelmach:
>>> It was <2015-04-30 czw 11:12>, when Richard Weinberger wrote:
 Am 30.04.2015 um 11:05 schrieb Łukasz Stelmach:
> Regardless, of initrd issues I feel there is a need of a local IPC
> that is more capable than UDS.
>>> [...]
>>> For example, a service can't aquire credentials of a client process that
>>> actually sent a request (it can, but it can't trust them). The service
>>> can't be protected by LSM on a bus that is driven by dbus-daemon. Yes,
>>> dbus-daemon, can check client's and srevice's labels and enforce a
>>> policy but it is going to be the daemon and not the LSM code in the
>>> kernel.
>>
>> That's why I said we can think of new kernel features if they are
>> needed.  But they current sink or swim approach of kdbus folks is also
>> not the solution.  As I said, if dbus-daemon utilizes the kernel
>> interface as much as possible we can think of new features.
>
> What kernel interfaces do you suggest to use to solve the issues
> I mentioned in the second paragraph: race conditions, LSM support (for
> example)?

 The question is whether it makes sense to collect this kind of meta data.
 I really like Andy and Alan's idea improve AF_UNIX or revive AF_BUS.
>>>
>>> Race conditions have nothing to do with metadata. Neither has LSM
>>> support.
>>
>> Sorry, I thought you mean the races while collecting metadata in userspace...
>
> My bad, some reace conditions *are* associated with collecting metadata
> but ont all. It is impossible (correct me if I am wrong) to implement
> reliable die-on-idle with dbus-daemon.
>
>>> AF_UNIX with multicast support wouldn't be AF_UNIX anymore.
>>>
>>> AF_BUS? I haven't followed the discussion back then. Why do you think it
>>> is better than kdbus?
>>
>> Please see https://lwn.net/Articles/641278/
>
> Thanks. If I understand correctly, the author suggests using EBPF on a
> receiveing socket side for receiving multicast messages. This is nice if
> you care about introducing (or not) (too?) much of new code. However,
> AFAICT it may be more computationally complex than Bloom filters because
> you need to run EBPF on every receiving socket instead of getting a list
> of a few of them to copy data to. Of course for small number of
> receivers the "constant" cost of running the Bloom filter may be higher.

Still think about the idea of implementing KDBUS in the form of socket.

What about using __multicast group__ instead of EBPF, to send/receive
multicast message?

(Which can implement the bloom filter as follows ?) E.g.

Sender:
send to multi_address

Receivers:
if ((multi_address & joined_address) == joined_address) {
/* a message for us */
}

Then we can further apply EBFP to remove the "False positive" case,
which will otherwise wake up user space code, and let it clear "False
positive" case.



-- 
Regards,

- cee1
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[cee1]

2015-04-30 22:52 GMT+08:00 Łukasz Stelmach l.stelm...@samsung.com:
 It was 2015-04-30 czw 14:45, when Richard Weinberger wrote:
 Am 30.04.2015 um 14:40 schrieb Łukasz Stelmach:
 It was 2015-04-30 czw 14:23, when Richard Weinberger wrote:
 Am 30.04.2015 um 14:16 schrieb Łukasz Stelmach:
 It was 2015-04-30 czw 12:40, when Richard Weinberger wrote:
 Am 30.04.2015 um 12:19 schrieb Łukasz Stelmach:
 It was 2015-04-30 czw 11:12, when Richard Weinberger wrote:
 Am 30.04.2015 um 11:05 schrieb Łukasz Stelmach:
 Regardless, of initrd issues I feel there is a need of a local IPC
 that is more capable than UDS.
 [...]
 For example, a service can't aquire credentials of a client process that
 actually sent a request (it can, but it can't trust them). The service
 can't be protected by LSM on a bus that is driven by dbus-daemon. Yes,
 dbus-daemon, can check client's and srevice's labels and enforce a
 policy but it is going to be the daemon and not the LSM code in the
 kernel.

 That's why I said we can think of new kernel features if they are
 needed.  But they current sink or swim approach of kdbus folks is also
 not the solution.  As I said, if dbus-daemon utilizes the kernel
 interface as much as possible we can think of new features.

 What kernel interfaces do you suggest to use to solve the issues
 I mentioned in the second paragraph: race conditions, LSM support (for
 example)?

 The question is whether it makes sense to collect this kind of meta data.
 I really like Andy and Alan's idea improve AF_UNIX or revive AF_BUS.

 Race conditions have nothing to do with metadata. Neither has LSM
 support.

 Sorry, I thought you mean the races while collecting metadata in userspace...

 My bad, some reace conditions *are* associated with collecting metadata
 but ont all. It is impossible (correct me if I am wrong) to implement
 reliable die-on-idle with dbus-daemon.

 AF_UNIX with multicast support wouldn't be AF_UNIX anymore.

 AF_BUS? I haven't followed the discussion back then. Why do you think it
 is better than kdbus?

 Please see https://lwn.net/Articles/641278/

 Thanks. If I understand correctly, the author suggests using EBPF on a
 receiveing socket side for receiving multicast messages. This is nice if
 you care about introducing (or not) (too?) much of new code. However,
 AFAICT it may be more computationally complex than Bloom filters because
 you need to run EBPF on every receiving socket instead of getting a list
 of a few of them to copy data to. Of course for small number of
 receivers the constant cost of running the Bloom filter may be higher.

Still think about the idea of implementing KDBUS in the form of socket.

What about using __multicast group__ instead of EBPF, to send/receive
multicast message?

(Which can implement the bloom filter as follows ?) E.g.

Sender:
send to multi_address

Receivers:
if ((multi_address  joined_address) == joined_address) {
/* a message for us */
}

Then we can further apply EBFP to remove the False positive case,
which will otherwise wake up user space code, and let it clear False
positive case.



-- 
Regards,

- cee1
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Jindřich Makovička]

On Mon, 27 Apr 2015 15:14:49 -0700
Linus Torvalds  wrote:

> On Mon, Apr 27, 2015 at 3:00 PM, Linus Torvalds
>  wrote:
> >
> > IOW, all the people who say that it's about avoiding context
> > switches are probably just full of shit. It's not about context
> > switches, it's about bad user-level code.
> 
> Just to make sure, I did a system-wide profile (so that you can
> actually see the overhead of context switching better), and that
> didn't change the picture.
> 
> The scheduler overhead *might* be 1% or so.
> 
> So really. The people who talk about how kdbus improves performance
> are just full of sh*t. Yes, it improves things, but the improvement
> seems to be 100% "incidental", in that it avoids a few trips down the
> user-space problems.
> 
> The real problems seem to be in dbus memory management (suggestion:
> keep a small per-thread cache of those message allocations) and to a
> smaller degree in the crazy utf8 validation (why the f*ck does it do
> that anyway?), with some locking problems thrown in for good measure.

In case someone actually still reads this, I guess the global rw_lock in
gobject/gtype.c, used by the GDbus binding, is one of the culprits.
Every GType instance allocation/ deallocation is serialized using this
lock, which pretty much disqualifies GObject from being used for
anything scalable to multiple threads.

GStreamer used to have serious performance issues due to that, which
AFAIK have been solved by removing GType from GStreamer core in the 1.0
release.

Regards,

-- 
Jindrich Makovicka
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Jiri Kosina]

On Mon, 22 Jun 2015, Jindrich Makovicka wrote:

> >> IOW, all the people who say that it's about avoiding context switches
> >> are probably just full of shit. It's not about context switches, it's
> >> about bad user-level code.
> > 
> > Just to make sure, I did a system-wide profile (so that you can actually
> > see the overhead of context switching better), and that didn't change
> > the picture.
> > 
> > The scheduler overhead *might* be 1% or so.
> > 
> > So really. The people who talk about how kdbus improves performance are
> > just full of sh*t. Yes, it improves things, but the improvement seems to
> > be 100% "incidental", in that it avoids a few trips down the user-space
> > problems.
> > 
> > The real problems seem to be in dbus memory management (suggestion: keep
> > a small per-thread cache of those message allocations) and to a smaller
> > degree in the crazy utf8 validation (why the f*ck does it do that
> > anyway?), with some locking problems thrown in for good measure.
> 
> In case someone actually still reads this, I guess the global rw_lock in 
> gobject/gtype.c is one of the culprits. Every GType instance allocation/
> deallocation is serialized using this lock, which pretty much 
> disqualifies GObject from being used for anything scalable to multiple 
> threads.
> 
> GStreamer used to have serious performance issues due to that, which 
> AFAIK have been solved by removing GType from GStreamer core in the 1.0 
> release.

This is interesting piece of information, but you unfortunately dropped 
everybody from CC, so it's very likely that it's going to be lost in the 
noise.

Could you please resend with CC list restored?

Thanks,

-- 
Jiri Kosina
SUSE Labs

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Jindrich Makovicka]

On Mon, 27 Apr 2015 15:14:49 -0700, Linus Torvalds wrote:

> On Mon, Apr 27, 2015 at 3:00 PM, Linus Torvalds
>  wrote:
>>
>> IOW, all the people who say that it's about avoiding context switches
>> are probably just full of shit. It's not about context switches, it's
>> about bad user-level code.
> 
> Just to make sure, I did a system-wide profile (so that you can actually
> see the overhead of context switching better), and that didn't change
> the picture.
> 
> The scheduler overhead *might* be 1% or so.
> 
> So really. The people who talk about how kdbus improves performance are
> just full of sh*t. Yes, it improves things, but the improvement seems to
> be 100% "incidental", in that it avoids a few trips down the user-space
> problems.
> 
> The real problems seem to be in dbus memory management (suggestion: keep
> a small per-thread cache of those message allocations) and to a smaller
> degree in the crazy utf8 validation (why the f*ck does it do that
> anyway?), with some locking problems thrown in for good measure.

In case someone actually still reads this, I guess the global rw_lock in 
gobject/gtype.c is one of the culprits. Every GType instance allocation/
deallocation is serialized using this lock, which pretty much 
disqualifies GObject from being used for anything scalable to multiple 
threads.

GStreamer used to have serious performance issues due to that, which 
AFAIK have been solved by removing GType from GStreamer core in the 1.0 
release.

Regards,
-- 
Jindrich Makovicka

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Jindrich Makovicka]

On Mon, 27 Apr 2015 15:14:49 -0700, Linus Torvalds wrote:

 On Mon, Apr 27, 2015 at 3:00 PM, Linus Torvalds
 torva...@linux-foundation.org wrote:

 IOW, all the people who say that it's about avoiding context switches
 are probably just full of shit. It's not about context switches, it's
 about bad user-level code.
 
 Just to make sure, I did a system-wide profile (so that you can actually
 see the overhead of context switching better), and that didn't change
 the picture.
 
 The scheduler overhead *might* be 1% or so.
 
 So really. The people who talk about how kdbus improves performance are
 just full of sh*t. Yes, it improves things, but the improvement seems to
 be 100% incidental, in that it avoids a few trips down the user-space
 problems.
 
 The real problems seem to be in dbus memory management (suggestion: keep
 a small per-thread cache of those message allocations) and to a smaller
 degree in the crazy utf8 validation (why the f*ck does it do that
 anyway?), with some locking problems thrown in for good measure.

In case someone actually still reads this, I guess the global rw_lock in 
gobject/gtype.c is one of the culprits. Every GType instance allocation/
deallocation is serialized using this lock, which pretty much 
disqualifies GObject from being used for anything scalable to multiple 
threads.

GStreamer used to have serious performance issues due to that, which 
AFAIK have been solved by removing GType from GStreamer core in the 1.0 
release.

Regards,
-- 
Jindrich Makovicka

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Jindřich Makovička]

On Mon, 27 Apr 2015 15:14:49 -0700
Linus Torvalds torva...@linux-foundation.org wrote:

 On Mon, Apr 27, 2015 at 3:00 PM, Linus Torvalds
 torva...@linux-foundation.org wrote:
 
  IOW, all the people who say that it's about avoiding context
  switches are probably just full of shit. It's not about context
  switches, it's about bad user-level code.
 
 Just to make sure, I did a system-wide profile (so that you can
 actually see the overhead of context switching better), and that
 didn't change the picture.
 
 The scheduler overhead *might* be 1% or so.
 
 So really. The people who talk about how kdbus improves performance
 are just full of sh*t. Yes, it improves things, but the improvement
 seems to be 100% incidental, in that it avoids a few trips down the
 user-space problems.
 
 The real problems seem to be in dbus memory management (suggestion:
 keep a small per-thread cache of those message allocations) and to a
 smaller degree in the crazy utf8 validation (why the f*ck does it do
 that anyway?), with some locking problems thrown in for good measure.

In case someone actually still reads this, I guess the global rw_lock in
gobject/gtype.c, used by the GDbus binding, is one of the culprits.
Every GType instance allocation/ deallocation is serialized using this
lock, which pretty much disqualifies GObject from being used for
anything scalable to multiple threads.

GStreamer used to have serious performance issues due to that, which
AFAIK have been solved by removing GType from GStreamer core in the 1.0
release.

Regards,

-- 
Jindrich Makovicka
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Jiri Kosina]

On Mon, 22 Jun 2015, Jindrich Makovicka wrote:

  IOW, all the people who say that it's about avoiding context switches
  are probably just full of shit. It's not about context switches, it's
  about bad user-level code.
  
  Just to make sure, I did a system-wide profile (so that you can actually
  see the overhead of context switching better), and that didn't change
  the picture.
  
  The scheduler overhead *might* be 1% or so.
  
  So really. The people who talk about how kdbus improves performance are
  just full of sh*t. Yes, it improves things, but the improvement seems to
  be 100% incidental, in that it avoids a few trips down the user-space
  problems.
  
  The real problems seem to be in dbus memory management (suggestion: keep
  a small per-thread cache of those message allocations) and to a smaller
  degree in the crazy utf8 validation (why the f*ck does it do that
  anyway?), with some locking problems thrown in for good measure.
 
 In case someone actually still reads this, I guess the global rw_lock in 
 gobject/gtype.c is one of the culprits. Every GType instance allocation/
 deallocation is serialized using this lock, which pretty much 
 disqualifies GObject from being used for anything scalable to multiple 
 threads.
 
 GStreamer used to have serious performance issues due to that, which 
 AFAIK have been solved by removing GType from GStreamer core in the 1.0 
 release.

This is interesting piece of information, but you unfortunately dropped 
everybody from CC, so it's very likely that it's going to be lost in the 
noise.

Could you please resend with CC list restored?

Thanks,

-- 
Jiri Kosina
SUSE Labs

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
Please read the FAQ at  http://www.tux.org/lkml/

Re: Sharing credentials in general (Re: [GIT PULL] kdbus for 4.1-rc1)

[Havoc Pennington]

On Fri, May 1, 2015 at 9:48 PM, Andy Lutomirski  wrote:
> Havoc, am I missing something here?  If I'm right about this aspect of
> D-Bus, then I'm a bit surprised.
>

I'm not well-informed about Binder, though from reading about it, it
seems to be modeled on and comparable to COM.

>From what I can tell Binder cannot be implemented in userspace, it
depends on a kernel piece for its semantics, so that's one reason we
wouldn't have made some of its design decisions. Should we make some
of them now if kernel is on the table - I don't know enough about
binder and kdbus to say, but interesting question.

When we did dbus, Android didn't exist so Binder was (I guess) just a
weird BeOS thing, I don't think I had ever looked at it.

COM however was top-of-mind. Mozilla had XPCOM, and GNOME's ORBit and
Bonobo stuff was COM-inspired.

The original idea of COM as far as I know wasn't IPC but more solving
a C++ ABI problem. C++ objects didn't have a fixed ABI (dependent on
compiler, which wasn't stable then) and C++ objects don't define
refcounting, introspection, and stuff like that. So COM provided a
more stable and complete C++ object model, and this evolved to (for
example) allow a COM interface to be used from multiple languages such
as Visual Basic. It makes C++ objects more like java.lang.Object or
GObject or other languages with more complete runtimes.

A COM object can be implemented by an in-proc dll or by a separate
process: http://blogs.msdn.com/b/larryosterman/archive/2004/10/12/241420.aspx

When you instantiate a COM object it's analogous to a C++ "new", it
isn't analogous to a dbus "service activation" - COM doesn't help you
with having a race-free singleton provider of a service, according to
http://stackoverflow.com/questions/6252334/singleton-com-object-required
anyway (and the one time I did win32 programming I remember getting a
singleton by creating a hidden window with a certain class name on it
or some hack like that, while on Linux for the same app we just used
dbus).

I don't think COM addresses the "multicast" thing either.

Anyway, if COM starts from "fancy C++ objects" and then happens to
have out-of-process mode, dbus starts from "singleton service
directory and multicasting" and then happens to have some
convention-only ways to map that onto native language objects.

dbus here was like X11 and KDE's DCOP, and moved away from GNOME's
ORBit and Mozilla's XPCOM.

To fill the COM-like role, GNOME ended up just extending GObject (such
as by adding full introspection).

I'm trying to remember the full rationale here, but some things I remember:

 * each language or framework already had the COM-like thing for
in-process: java.lang.Object, GObject, python object, etc. These
define refcounting or GC, introspection, and all that stuff already.
App developers generally want to work with their "native" object
runtime.

 * the abstraction where objects could be either in or out of process
wasn't that useful; in practice each object was always one or the
other, and had to be treated differently depending on which.
Philosophically, "network transparent objects" as promised by CORBA
were not working for us.

 * multicast and singleton services were really useful, while
random-object-happens-to-be-out-of-process really wasn't. The usual
reason to go out of process is because there's some sort of shared
resource (like "the user's task bar" or "the wifi connection"), so
singleton is the default case. There are two common singleton
"scopes," per-machine (system bus) and per-session (user bus).

 * cross-process reference counting made a big mess; without the
hub-and-spoke as in dbus, it's either unreliable or wastes a lot of
file descriptors as every process tracks every other process via an
open connection. (binder solves this presumably via the kernel, but we
were looking at userspace-only options.)

 * recursion never had a satisfactory solution; you'd call some
innocuous-looking method on what turned out to be, or use, a remote
object and suddenly code from some completely unrelated part of your
app might end up running. so you call foo_ref() and while that ref is
going to another process you get an incoming unref on another object
you were using... needless to say these bugs were always heisenbugs...
essentially it was multithreaded programming but without locks. The
idea with dbus is to make explicitly asynchronous remote calls kind of
the default, or at least well-supported, though language bindings can
always choose otherwise. In dbus if you make a blocking remote call,
traditionally it is truly blocking (though it's up to the language
binding). The thread making the blocking call will not process any
incoming recursive calls. If you get a deadlock, you need to make your
code nonblocking, which you probably should have done in the first
place.
http://lists.freedesktop.org/archives/dbus/2007-March/007381.html and
http://lists.freedesktop.org/archives/dbus/2006-March/004362.html are
some sample 

Re: Sharing credentials in general (Re: [GIT PULL] kdbus for 4.1-rc1)

[Havoc Pennington]

On Fri, May 1, 2015 at 9:48 PM, Andy Lutomirski l...@amacapital.net wrote:
 Havoc, am I missing something here?  If I'm right about this aspect of
 D-Bus, then I'm a bit surprised.


I'm not well-informed about Binder, though from reading about it, it
seems to be modeled on and comparable to COM.

From what I can tell Binder cannot be implemented in userspace, it
depends on a kernel piece for its semantics, so that's one reason we
wouldn't have made some of its design decisions. Should we make some
of them now if kernel is on the table - I don't know enough about
binder and kdbus to say, but interesting question.

When we did dbus, Android didn't exist so Binder was (I guess) just a
weird BeOS thing, I don't think I had ever looked at it.

COM however was top-of-mind. Mozilla had XPCOM, and GNOME's ORBit and
Bonobo stuff was COM-inspired.

The original idea of COM as far as I know wasn't IPC but more solving
a C++ ABI problem. C++ objects didn't have a fixed ABI (dependent on
compiler, which wasn't stable then) and C++ objects don't define
refcounting, introspection, and stuff like that. So COM provided a
more stable and complete C++ object model, and this evolved to (for
example) allow a COM interface to be used from multiple languages such
as Visual Basic. It makes C++ objects more like java.lang.Object or
GObject or other languages with more complete runtimes.

A COM object can be implemented by an in-proc dll or by a separate
process: http://blogs.msdn.com/b/larryosterman/archive/2004/10/12/241420.aspx

When you instantiate a COM object it's analogous to a C++ new, it
isn't analogous to a dbus service activation - COM doesn't help you
with having a race-free singleton provider of a service, according to
http://stackoverflow.com/questions/6252334/singleton-com-object-required
anyway (and the one time I did win32 programming I remember getting a
singleton by creating a hidden window with a certain class name on it
or some hack like that, while on Linux for the same app we just used
dbus).

I don't think COM addresses the multicast thing either.

Anyway, if COM starts from fancy C++ objects and then happens to
have out-of-process mode, dbus starts from singleton service
directory and multicasting and then happens to have some
convention-only ways to map that onto native language objects.

dbus here was like X11 and KDE's DCOP, and moved away from GNOME's
ORBit and Mozilla's XPCOM.

To fill the COM-like role, GNOME ended up just extending GObject (such
as by adding full introspection).

I'm trying to remember the full rationale here, but some things I remember:

 * each language or framework already had the COM-like thing for
in-process: java.lang.Object, GObject, python object, etc. These
define refcounting or GC, introspection, and all that stuff already.
App developers generally want to work with their native object
runtime.

 * the abstraction where objects could be either in or out of process
wasn't that useful; in practice each object was always one or the
other, and had to be treated differently depending on which.
Philosophically, network transparent objects as promised by CORBA
were not working for us.

 * multicast and singleton services were really useful, while
random-object-happens-to-be-out-of-process really wasn't. The usual
reason to go out of process is because there's some sort of shared
resource (like the user's task bar or the wifi connection), so
singleton is the default case. There are two common singleton
scopes, per-machine (system bus) and per-session (user bus).

 * cross-process reference counting made a big mess; without the
hub-and-spoke as in dbus, it's either unreliable or wastes a lot of
file descriptors as every process tracks every other process via an
open connection. (binder solves this presumably via the kernel, but we
were looking at userspace-only options.)

 * recursion never had a satisfactory solution; you'd call some
innocuous-looking method on what turned out to be, or use, a remote
object and suddenly code from some completely unrelated part of your
app might end up running. so you call foo_ref() and while that ref is
going to another process you get an incoming unref on another object
you were using... needless to say these bugs were always heisenbugs...
essentially it was multithreaded programming but without locks. The
idea with dbus is to make explicitly asynchronous remote calls kind of
the default, or at least well-supported, though language bindings can
always choose otherwise. In dbus if you make a blocking remote call,
traditionally it is truly blocking (though it's up to the language
binding). The thread making the blocking call will not process any
incoming recursive calls. If you get a deadlock, you need to make your
code nonblocking, which you probably should have done in the first
place.
http://lists.freedesktop.org/archives/dbus/2007-March/007381.html and
http://lists.freedesktop.org/archives/dbus/2006-March/004362.html are
some sample 

Re: Sharing credentials in general (Re: [GIT PULL] kdbus for 4.1-rc1)

[Andy Lutomirski]

On Mon, Apr 27, 2015 at 9:33 AM, David Herrmann  wrote:
> On Mon, Apr 27, 2015 at 6:13 PM, Andy Lutomirski  wrote:
>> 2.  This is a nice thought, but it doesn't work in practice.  Sorry.
>> I can give you a big pile of CVEs from last year if you like, or I can
>> try explaining again.
>>
>> The issue boils down to what type of privileges you want to assert and
>> over what object you want to assert them.  Suppose I have a method
>> "write".  When I call it, I do Write(destination, text).  In your
>> model, it's basically never safe to do:
>
> You're correct. So don't create such APIs.
> In fact, never ever accept FDs or file-paths from a less privileged
> caller. It might be FUSE backed and under their full control.

Really?

So writing to stdout is not okay?  So what if it's on FUSE?  At worst
it should be a DoS, unless you screw up the security model (as the
kernel did) and it's privilege escalation or system state corruption.

How about privileged services that mediate interactions between
client-provided objects and other client-inaccessible things?

FWIW, all these threads seem to have spawned tons on comparisons
between dbus, COM, Binder, etc.  I don't know too much about all of
these technologies, but one difference springs to mind.  COM and
Binder both have very clear concepts of capability-passing.  COM calls
it "marshalling an interface".  Binder calls it "IBinder".  Quoting
from the reference [1]:

The data sent through transact() is a Parcel, a generic buffer of data
that also maintains some meta-data about its contents. The meta data
is used to manage IBinder object references in the buffer, so that
those references can be maintained as the buffer moves across
processes.

Wikipedia suggests CORBA has "objects by reference".  I think that
Wayland's "new_id" type is effectively an object reference.  I have no
clue what Mac OS X and iOS do.

AFAICT D-Bus is completely missing any corresponding concept.  I'm
rather confused how Samsung thinks they can cleanly layer Binder over
kdbus unless they use kdbus as a completely dumb transport and
implement object references in userspace.  If they do that, then I
don't see the point of using kdbus at all -- sockets would be fine.

One can certainly debate the merits of capability-based security, but
I've (so far) never heard anyone claim that passing object references
around is a bad idea.

Havoc, am I missing something here?  If I'm right about this aspect of
D-Bus, then I'm a bit surprised.

[1] http://developer.android.com/reference/android/os/IBinder.html

--Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Austin S Hemmelgarn]

On 2015-04-29 08:47, Harald Hoyer wrote:

Until then the whole common IPC problem is unresolved and Linux
distributions are just a collection of random software with no common
interoperability and home grown interfaces.
I don't know how I managed to not notice this comment before, but I find 
it particularly hilarious.
The part about 'no common interoperability' is just plain BS with the 
exception of some of the insanity being touted by systemd advocates and 
the insanity that is accessibility software on linux, you can easily 
string together pretty much arbitrary strings of commands using fifo's 
to achieve almost anything; the actual interoperability issues (WRT to 
the command line at least, which is where all the stuff you are 
complaining about works) come up only with stuff (like journald for 
example) that just refuses to use text interfaces on the command-line.
Also, the 'home grown interfaces' you are complaining about are used on 
every operating system (not just Linux or other Unix progeny) every day, 
and no amount of better IPC is going to stop that; furthermore, almost 
every current 'standard' protocol or interface used on the internet 
started out as a 'home grown interface' (TCP/IP immediately comes to 
mind, followed shortly by NFS, SMTP, WebDAV, SSH, XML, JSON, and a whole 
slew of others).




smime.p7s
Description: S/MIME Cryptographic Signature

Re: Sharing credentials in general (Re: [GIT PULL] kdbus for 4.1-rc1)

[Andy Lutomirski]

On Mon, Apr 27, 2015 at 9:33 AM, David Herrmann dh.herrm...@gmail.com wrote:
 On Mon, Apr 27, 2015 at 6:13 PM, Andy Lutomirski l...@amacapital.net wrote:
 2.  This is a nice thought, but it doesn't work in practice.  Sorry.
 I can give you a big pile of CVEs from last year if you like, or I can
 try explaining again.

 The issue boils down to what type of privileges you want to assert and
 over what object you want to assert them.  Suppose I have a method
 write.  When I call it, I do Write(destination, text).  In your
 model, it's basically never safe to do:

 You're correct. So don't create such APIs.
 In fact, never ever accept FDs or file-paths from a less privileged
 caller. It might be FUSE backed and under their full control.

Really?

So writing to stdout is not okay?  So what if it's on FUSE?  At worst
it should be a DoS, unless you screw up the security model (as the
kernel did) and it's privilege escalation or system state corruption.

How about privileged services that mediate interactions between
client-provided objects and other client-inaccessible things?

FWIW, all these threads seem to have spawned tons on comparisons
between dbus, COM, Binder, etc.  I don't know too much about all of
these technologies, but one difference springs to mind.  COM and
Binder both have very clear concepts of capability-passing.  COM calls
it marshalling an interface.  Binder calls it IBinder.  Quoting
from the reference [1]:

The data sent through transact() is a Parcel, a generic buffer of data
that also maintains some meta-data about its contents. The meta data
is used to manage IBinder object references in the buffer, so that
those references can be maintained as the buffer moves across
processes.

Wikipedia suggests CORBA has objects by reference.  I think that
Wayland's new_id type is effectively an object reference.  I have no
clue what Mac OS X and iOS do.

AFAICT D-Bus is completely missing any corresponding concept.  I'm
rather confused how Samsung thinks they can cleanly layer Binder over
kdbus unless they use kdbus as a completely dumb transport and
implement object references in userspace.  If they do that, then I
don't see the point of using kdbus at all -- sockets would be fine.

One can certainly debate the merits of capability-based security, but
I've (so far) never heard anyone claim that passing object references
around is a bad idea.

Havoc, am I missing something here?  If I'm right about this aspect of
D-Bus, then I'm a bit surprised.

[1] http://developer.android.com/reference/android/os/IBinder.html

--Andy
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Austin S Hemmelgarn]

On 2015-04-29 08:47, Harald Hoyer wrote:

Until then the whole common IPC problem is unresolved and Linux
distributions are just a collection of random software with no common
interoperability and home grown interfaces.
I don't know how I managed to not notice this comment before, but I find 
it particularly hilarious.
The part about 'no common interoperability' is just plain BS with the 
exception of some of the insanity being touted by systemd advocates and 
the insanity that is accessibility software on linux, you can easily 
string together pretty much arbitrary strings of commands using fifo's 
to achieve almost anything; the actual interoperability issues (WRT to 
the command line at least, which is where all the stuff you are 
complaining about works) come up only with stuff (like journald for 
example) that just refuses to use text interfaces on the command-line.
Also, the 'home grown interfaces' you are complaining about are used on 
every operating system (not just Linux or other Unix progeny) every day, 
and no amount of better IPC is going to stop that; furthermore, almost 
every current 'standard' protocol or interface used on the internet 
started out as a 'home grown interface' (TCP/IP immediately comes to 
mind, followed shortly by NFS, SMTP, WebDAV, SSH, XML, JSON, and a whole 
slew of others).




smime.p7s
Description: S/MIME Cryptographic Signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Eric W. Biederman]

On April 29, 2015 7:47:53 AM CDT, Harald Hoyer  wrote:
>-BEGIN PGP SIGNED MESSAGE-
>Hash: SHA256
>
>On 29.04.2015 01:12, John Stoffel wrote:
>> LDAP is pretty damn generic, in that you can put pretty large objects
>into
>> it, and pretty large OUs, etc.  So why would it be a candidate for
>going
>> into the kernel?  And why is kdbus so important in the kernel as
>well?
>> People have talked about it needing to be there for bootup, but isn't
>that
>> why we ripped out RAID detection and such from the kernel and built
>> initramfs, so that there's LESS in the kernel, and more in an early
>> userspace?  Same idea with dbus in my opinion.
>
>Let me elaborate on the initramfs/shutdown situation a little bit more,
>because I have to deal with that every day.
>
>Because of the "let's move everything to userspace" sentiment we
>nowadays
>have the situation, that we need a lot of tools to setup the root
>device.
>
>Be it LVM on IMSM or iSCSI multipath, the initramfs has to setup the
>network
>(with bridging, bonding, etc.), the iSCSI connection, assemble the
>raid, the
>LVM, open crypto devices, etc...
>And if something goes wrong, you want to have a shell, see all the logs
>and
>debug things.
>
>Now over the time we moved away from simple shell scripts (without any
>logging) and static compiled special versions for the initramfs to a
>mini
>distribution in the initramfs, which simplifies maintenance and
>improves
>reliability.
>
>Basically you want to use the same tools in the initramfs (and
>shutdown)
>which you already have and use in your real root, with the same
>configuration
>files and the same interfaces and the same code paths.
>
>Therefore systemd is started in dracut created initramfs, which starts
>journald for logging. The same basic systemd targets exist in the
>initramfs
>as on the real root, so normally you don't have to cope with
>specialized
>versions for the initramfs.
>
>The target here is to have the same IPC mechanism from the very
>beginning to
>the very end. No crappy fallback mechanisms in case a daemon is not
>running
>or has crashed, no creepy transition from initramfs root to real root
>to
>shutdown root.
>
>We already have such transitions like: systemd, journald, mdmon [1],
>etc.
>systemd has to serialize itself, journald's file descriptors are
>transitioned
>over, mdmon jumps through hoops. Remember you want to get rid of open
>files
>and executables and have to reexec everything, if you transition from
>the
>initramfs root to the real root, and also from the real root to the
>shutdown
>root.
>
>We really don't want the IPC mechanism to be in a flux state. All tools
>have
>to fallback to a non-standard mechanism in that case.
>
>If I have to pull in a dbus daemon in the initramfs, we still have the
>chicken and egg problem for PID 1 talking to the logging daemon and
>starting
>dbus.
>systemd cannot talk to journald via dbus unless dbus-daemon is started,
>dbus
>cannot log anything on startup, if journald is not running, etc...
>
>dbus-daemon would have to transition to the real root, and from the
>real root
>to the shutdown root, without losing state.

Which does not sound fundamentally hard.  
Unify the roots, and make /run or wherever the dbus socket lives always 
available.  As long as your initramfs has the latest versions of software there 
is no need for any tricky transitions except to upgrade software on a running 
system.


>Of course this can all be done, but it would involve fallback
>mechanisms,
>which we want to get rid off. 

Only if you design things poorly.

>Hopefully, you don't suggest to merge
>dbus with
>PID 1. Also with a daemon, you will lose the points mentioned in the
>cover mail

I don't see how something that is inappropriate to be in PID 1 is better in PID 
0.


>I don't care, if the kdbus speedup is only marginal.
>
>In my ideal world, there is a standard IPC mechanism from the beginning
>to
>the end, which does not rely on any process running (except the kernel)
>and
>which is used by _all_ tools, be it a system daemon providing
>information and
>interfaces about device assembly or network setup tools or end user
>desktop
>processes.

And that is a beautiful dream and an absolutely rubbish way to get there.  If 
the performance is not top notch everything can not use your beautiful IPC 
mechanism.Which means your dream fails.

Good performance is a hard requirement to get where you want to be.

>dbus _is_ such an easy, flexible standard IPC mechanism. Of course, you
>can
>invent the wheel again (NIH, "we know better") and wait and see, if
>that
>works out. Until then the whole common IPC problem is unresolved and
>Linux
>distributions are just a collection of random software with no common
>interoperability and home grown interfaces.


kdbus seems to be the NIH "we know better better" approach.  Many of it's 
design decisions we have chosen to differently elsewhere in the kernel because 
the have caused problems.  When these issues have been 

Re: [GIT PULL] kdbus for 4.1-rc1

[Richard Weinberger]

Am 30.04.2015 um 16:52 schrieb Łukasz Stelmach:
>> Sorry, I thought you mean the races while collecting metadata in userspace...
> 
> My bad, some reace conditions *are* associated with collecting metadata
> but ont all. It is impossible (correct me if I am wrong) to implement
> reliable die-on-idle with dbus-daemon.

IIRC Andy gave some ideas howto deal with that.
i.e. https://lkml.org/lkml/2015/4/29/622

>>> AF_UNIX with multicast support wouldn't be AF_UNIX anymore.
>>>
>>> AF_BUS? I haven't followed the discussion back then. Why do you think it
>>> is better than kdbus?
>>
>> Please see https://lwn.net/Articles/641278/
> 
> Thanks. If I understand correctly, the author suggests using EBPF on a
> receiveing socket side for receiving multicast messages. This is nice if
> you care about introducing (or not) (too?) much of new code. However,
> AFAICT it may be more computationally complex than Bloom filters because
> you need to run EBPF on every receiving socket instead of getting a list
> of a few of them to copy data to. Of course for small number of
> receivers the "constant" cost of running the Bloom filter may be higher.

To make the story short, the kdbus *concept* needs much more
thought. There are many ideas out there howto deal with dbus issues without 
introducing
an ad-hoc solution. AF_BUS is just one of them.
IMHO AF_BUS would be nice but the decision is not up to me.

Thanks,
//richard



signature.asc
Description: OpenPGP digital signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Łukasz Stelmach]

It was <2015-04-30 czw 14:45>, when Richard Weinberger wrote:
> Am 30.04.2015 um 14:40 schrieb Łukasz Stelmach:
>> It was <2015-04-30 czw 14:23>, when Richard Weinberger wrote:
>>> Am 30.04.2015 um 14:16 schrieb Łukasz Stelmach:
 It was <2015-04-30 czw 12:40>, when Richard Weinberger wrote:
> Am 30.04.2015 um 12:19 schrieb Łukasz Stelmach:
>> It was <2015-04-30 czw 11:12>, when Richard Weinberger wrote:
>>> Am 30.04.2015 um 11:05 schrieb Łukasz Stelmach:
 Regardless, of initrd issues I feel there is a need of a local IPC
 that is more capable than UDS. 
>> [...]
>> For example, a service can't aquire credentials of a client process that
>> actually sent a request (it can, but it can't trust them). The service
>> can't be protected by LSM on a bus that is driven by dbus-daemon. Yes,
>> dbus-daemon, can check client's and srevice's labels and enforce a
>> policy but it is going to be the daemon and not the LSM code in the
>> kernel.
>
> That's why I said we can think of new kernel features if they are
> needed.  But they current sink or swim approach of kdbus folks is also
> not the solution.  As I said, if dbus-daemon utilizes the kernel
> interface as much as possible we can think of new features.

 What kernel interfaces do you suggest to use to solve the issues
 I mentioned in the second paragraph: race conditions, LSM support (for
 example)?
>>>
>>> The question is whether it makes sense to collect this kind of meta data.
>>> I really like Andy and Alan's idea improve AF_UNIX or revive AF_BUS.
>> 
>> Race conditions have nothing to do with metadata. Neither has LSM
>> support.
>
> Sorry, I thought you mean the races while collecting metadata in userspace...

My bad, some reace conditions *are* associated with collecting metadata
but ont all. It is impossible (correct me if I am wrong) to implement
reliable die-on-idle with dbus-daemon.

>> AF_UNIX with multicast support wouldn't be AF_UNIX anymore.
>> 
>> AF_BUS? I haven't followed the discussion back then. Why do you think it
>> is better than kdbus?
>
> Please see https://lwn.net/Articles/641278/

Thanks. If I understand correctly, the author suggests using EBPF on a
receiveing socket side for receiving multicast messages. This is nice if
you care about introducing (or not) (too?) much of new code. However,
AFAICT it may be more computationally complex than Bloom filters because
you need to run EBPF on every receiving socket instead of getting a list
of a few of them to copy data to. Of course for small number of
receivers the "constant" cost of running the Bloom filter may be higher.

Kind regards,
-- 
Łukasz Stelmach
Samsung R Institute Poland
Samsung Electronics


signature.asc
Description: PGP signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Richard Weinberger]

Am 30.04.2015 um 14:40 schrieb Łukasz Stelmach:
> It was <2015-04-30 czw 14:23>, when Richard Weinberger wrote:
>> Am 30.04.2015 um 14:16 schrieb Łukasz Stelmach:
>>> It was <2015-04-30 czw 12:40>, when Richard Weinberger wrote:
 Am 30.04.2015 um 12:19 schrieb Łukasz Stelmach:
> It was <2015-04-30 czw 11:12>, when Richard Weinberger wrote:
>> Am 30.04.2015 um 11:05 schrieb Łukasz Stelmach:
>>> Regardless, of initrd issues I feel there is a need of a local IPC
>>> that is more capable than UDS. 
> [...]
> For example, a service can't aquire credentials of a client process that
> actually sent a request (it can, but it can't trust them). The service
> can't be protected by LSM on a bus that is driven by dbus-daemon. Yes,
> dbus-daemon, can check client's and srevice's labels and enforce a
> policy but it is going to be the daemon and not the LSM code in the
> kernel.

 That's why I said we can think of new kernel features if they are
 needed.  But they current sink or swim approach of kdbus folks is also
 not the solution.  As I said, if dbus-daemon utilizes the kernel
 interface as much as possible we can think of new features.
>>>
>>> What kernel interfaces do you suggest to use to solve the issues
>>> I mentioned in the second paragraph: race conditions, LSM support (for
>>> example)?
>>
>> The question is whether it makes sense to collect this kind of meta data.
>> I really like Andy and Alan's idea improve AF_UNIX or revive AF_BUS.
> 
> Race conditions have nothing to do with metadata. Neither has LSM
> support.

Sorry, I thought you mean the races while collecting metadata in userspace...

> AF_UNIX with multicast support wouldn't be AF_UNIX anymore.
> 
> AF_BUS? I haven't followed the discussion back then. Why do you think it
> is better than kdbus?

Please see https://lwn.net/Articles/641278/

Thanks,
//richard



signature.asc
Description: OpenPGP digital signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Łukasz Stelmach]

It was <2015-04-30 czw 14:23>, when Richard Weinberger wrote:
> Am 30.04.2015 um 14:16 schrieb Łukasz Stelmach:
>> It was <2015-04-30 czw 12:40>, when Richard Weinberger wrote:
>>> Am 30.04.2015 um 12:19 schrieb Łukasz Stelmach:
 It was <2015-04-30 czw 11:12>, when Richard Weinberger wrote:
> Am 30.04.2015 um 11:05 schrieb Łukasz Stelmach:
>> Regardless, of initrd issues I feel there is a need of a local IPC
>> that is more capable than UDS. 
[...]
 For example, a service can't aquire credentials of a client process that
 actually sent a request (it can, but it can't trust them). The service
 can't be protected by LSM on a bus that is driven by dbus-daemon. Yes,
 dbus-daemon, can check client's and srevice's labels and enforce a
 policy but it is going to be the daemon and not the LSM code in the
 kernel.
>>>
>>> That's why I said we can think of new kernel features if they are
>>> needed.  But they current sink or swim approach of kdbus folks is also
>>> not the solution.  As I said, if dbus-daemon utilizes the kernel
>>> interface as much as possible we can think of new features.
>> 
>> What kernel interfaces do you suggest to use to solve the issues
>> I mentioned in the second paragraph: race conditions, LSM support (for
>> example)?
>
> The question is whether it makes sense to collect this kind of meta data.
> I really like Andy and Alan's idea improve AF_UNIX or revive AF_BUS.

Race conditions have nothing to do with metadata. Neither has LSM
support.

AF_UNIX with multicast support wouldn't be AF_UNIX anymore.

AF_BUS? I haven't followed the discussion back then. Why do you think it
is better than kdbus?

-- 
Łukasz Stelmach
Samsung R Institute Poland
Samsung Electronics


signature.asc
Description: PGP signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Richard Weinberger]

Am 30.04.2015 um 14:16 schrieb Łukasz Stelmach:
> It was <2015-04-30 czw 12:40>, when Richard Weinberger wrote:
>> Am 30.04.2015 um 12:19 schrieb Łukasz Stelmach:
>>> It was <2015-04-30 czw 11:12>, when Richard Weinberger wrote:
 Am 30.04.2015 um 11:05 schrieb Łukasz Stelmach:
> Regardless, of initrd issues I feel there is a need of a local IPC
> that is more capable than UDS. Linus Torvalds is probably right that
> dbus-daemon is everything but effictient. I disagree, however, that
> it can be optimised and therefore solve *all* issues kdbus is trying
> to address. dbus-deamon, by design, can't some things. It can't
> transmitt large payloads without copying them. It can't be made
> race-free.

 This is true.
 But as long dbus-deamon is not optimized as much as possible there is
 no reason to force push kdbus.
 As soon dbus-deamon exploits all kernel interfaces as much it can and
 it still needs work (may it performance or other stuff) we can think
 of new kernel features which can help dbus-deamon.
>>>
>>> I may not be well informed about kernel interfaces, but there are
>>> some use cases no dbus-daemon optimisation can make work properly
>>> because of rece-conditons introduced by the user-space based message
>>> router.
>>>
>>> For example, a service can't aquire credentials of a client process that
>>> actually sent a request (it can, but it can't trust them). The service
>>> can't be protected by LSM on a bus that is driven by dbus-daemon. Yes,
>>> dbus-daemon, can check client's and srevice's labels and enforce a
>>> policy but it is going to be the daemon and not the LSM code in the
>>> kernel.
>>
>> That's why I said we can think of new kernel features if they are
>> needed.  But they current sink or swim approach of kdbus folks is also
>> not the solution.  As I said, if dbus-daemon utilizes the kernel
>> interface as much as possible we can think of new features.
> 
> What kernel interfaces do you suggest to use to solve the issues
> I mentioned in the second paragraph: race conditions, LSM support (for
> example)?

The question is whether it makes sense to collect this kind of meta data.
I really like Andy and Alan's idea improve AF_UNIX or revive AF_BUS.

Thanks,
//richard



signature.asc
Description: OpenPGP digital signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Łukasz Stelmach]

It was <2015-04-30 czw 12:40>, when Richard Weinberger wrote:
> Am 30.04.2015 um 12:19 schrieb Łukasz Stelmach:
>> It was <2015-04-30 czw 11:12>, when Richard Weinberger wrote:
>>> Am 30.04.2015 um 11:05 schrieb Łukasz Stelmach:
 Regardless, of initrd issues I feel there is a need of a local IPC
 that is more capable than UDS. Linus Torvalds is probably right that
 dbus-daemon is everything but effictient. I disagree, however, that
 it can be optimised and therefore solve *all* issues kdbus is trying
 to address. dbus-deamon, by design, can't some things. It can't
 transmitt large payloads without copying them. It can't be made
 race-free.
>>>
>>> This is true.
>>> But as long dbus-deamon is not optimized as much as possible there is
>>> no reason to force push kdbus.
>>> As soon dbus-deamon exploits all kernel interfaces as much it can and
>>> it still needs work (may it performance or other stuff) we can think
>>> of new kernel features which can help dbus-deamon.
>> 
>> I may not be well informed about kernel interfaces, but there are
>> some use cases no dbus-daemon optimisation can make work properly
>> because of rece-conditons introduced by the user-space based message
>> router.
>> 
>> For example, a service can't aquire credentials of a client process that
>> actually sent a request (it can, but it can't trust them). The service
>> can't be protected by LSM on a bus that is driven by dbus-daemon. Yes,
>> dbus-daemon, can check client's and srevice's labels and enforce a
>> policy but it is going to be the daemon and not the LSM code in the
>> kernel.
>
> That's why I said we can think of new kernel features if they are
> needed.  But they current sink or swim approach of kdbus folks is also
> not the solution.  As I said, if dbus-daemon utilizes the kernel
> interface as much as possible we can think of new features.

What kernel interfaces do you suggest to use to solve the issues
I mentioned in the second paragraph: race conditions, LSM support (for
example)?

BTW. Does anyone know how microkernel-based OSes implement sockets?
-- 
Łukasz Stelmach
Samsung R Institute Poland
Samsung Electronics


signature.asc
Description: PGP signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Richard Weinberger]

Am 30.04.2015 um 12:19 schrieb Łukasz Stelmach:
> It was <2015-04-30 czw 11:12>, when Richard Weinberger wrote:
>> Am 30.04.2015 um 11:05 schrieb Łukasz Stelmach:
>>> Regardless, of initrd issues I feel there is a need of a local IPC
>>> that is more capable than UDS. Linus Torvalds is probably right that
>>> dbus-daemon is everything but effictient. I disagree, however, that
>>> it can be optimised and therefore solve *all* issues kdbus is trying
>>> to address. dbus-deamon, by design, can't some things. It can't
>>> transmitt large payloads without copying them. It can't be made
>>> race-free.
>>
>> This is true.
>> But as long dbus-deamon is not optimized as much as possible there is
>> no reason to force push kdbus.
>> As soon dbus-deamon exploits all kernel interfaces as much it can and
>> it still needs work (may it performance or other stuff) we can think
>> of new kernel features which can help dbus-deamon.
> 
> I may not be well informed about kernel interfaces, but there are some
> use cases no dbus-daemon optimisation can make work properly because of
> rece-conditons introduced by the user-space based message router.
> 
> For example, a service can't aquire credentials of a client process that
> actually sent a request (it can, but it can't trust them). The service
> can't be protected by LSM on a bus that is driven by dbus-daemon. Yes,
> dbus-daemon, can check client's and srevice's labels and enforce a
> policy but it is going to be the daemon and not the LSM code in the
> kernel.

That's why I said we can think of new kernel features if they are needed.
But they current sink or swim approach of kdbus folks is also not the solution.
As I said, if dbus-daemon utilizes the kernel interface as much as possible we
can think of new features.

Thanks,
//richard



signature.asc
Description: OpenPGP digital signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Łukasz Stelmach]

It was <2015-04-30 czw 11:12>, when Richard Weinberger wrote:
> Am 30.04.2015 um 11:05 schrieb Łukasz Stelmach:
>> Regardless, of initrd issues I feel there is a need of a local IPC
>> that is more capable than UDS. Linus Torvalds is probably right that
>> dbus-daemon is everything but effictient. I disagree, however, that
>> it can be optimised and therefore solve *all* issues kdbus is trying
>> to address. dbus-deamon, by design, can't some things. It can't
>> transmitt large payloads without copying them. It can't be made
>> race-free.
>
> This is true.
> But as long dbus-deamon is not optimized as much as possible there is
> no reason to force push kdbus.
> As soon dbus-deamon exploits all kernel interfaces as much it can and
> it still needs work (may it performance or other stuff) we can think
> of new kernel features which can help dbus-deamon.

I may not be well informed about kernel interfaces, but there are some
use cases no dbus-daemon optimisation can make work properly because of
rece-conditons introduced by the user-space based message router.

For example, a service can't aquire credentials of a client process that
actually sent a request (it can, but it can't trust them). The service
can't be protected by LSM on a bus that is driven by dbus-daemon. Yes,
dbus-daemon, can check client's and srevice's labels and enforce a
policy but it is going to be the daemon and not the LSM code in the
kernel.

-- 
Łukasz Stelmach
Samsung R Institute Poland
Samsung Electronics


signature.asc
Description: PGP signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Richard Weinberger]

Am 30.04.2015 um 11:05 schrieb Łukasz Stelmach:
> Regardless, of initrd issues I feel there is a need of a local IPC that
> is more capable than UDS. Linus Torvalds is probably right that
> dbus-daemon is everything but effictient. I disagree, however, that it
> can be optimised and therefore solve *all* issues kdbus is trying to
> address. dbus-deamon, by design, can't some things. It can't transmitt
> large payloads without copying them. It can't be made race-free.

This is true.
But as long dbus-deamon is not optimized as much as possible there is no reason
to force push kdbus.
As soon dbus-deamon exploits all kernel interfaces as much it can and it still 
needs
work (may it performance or other stuff) we can think of new kernel features 
which
can help dbus-deamon.

Thanks,
//richard



signature.asc
Description: OpenPGP digital signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Łukasz Stelmach]

It was <2015-04-29 śro 17:21>, when Austin S Hemmelgarn wrote:
> On 2015-04-29 11:03, Theodore Ts'o wrote:
>> On Wed, Apr 29, 2015 at 04:53:53PM +0200, Harald Hoyer wrote:
>>> Sure, I can write one binary to rule them all, pull out all the code from 
>>> all
>>> tools I need, but for me an IPC mechanism sounds a lot better. And it 
>>> should be
>>> _one_ common IPC mechanism and not a plethora of them. It should feel like 
>>> an
>>> operating system and not like a bunch of thrown together software, which is
>>> glued together with some magic shell scripts.
>>
>> And so requiring wireshark (and X?) in initramfs to debug problems
>> once dbus is introduced is better?
>>
>> I would think shell scripts are *easier* to debug when things go
>> wrong,
[...]
> I keep hearing from people that shell scripting is hard, it really
> isn't compared to a number of other scripting languages, you just need
> to actually learn to do it right (which is getting more and more
> difficult these days cause fewer and fewer CS schools are teaching
> Unix).

My 2/100 of a currency of your choice.

As much as I like(ed) shell scripts as a boot up tool and disliked
obscure boot-up procedures of some operating system, I can't help but
notice that GNU/Linux distributions have become very
sophisticated/complcated (cross out if not applicable). Personally
I feel that this degree of coplexity can't be supported by shell scripts
piping data around. It does not scale. I am not 100% sure a new IPC is
the answer, simply because I do not have experience to be so. It
definitely can be and the problem, as I see it, is real.

(The alternative answer is PowerShells capability to pipe objects. I
don't like it and I thik it's not a full answer.)

Regardless, of initrd issues I feel there is a need of a local IPC that
is more capable than UDS. Linus Torvalds is probably right that
dbus-daemon is everything but effictient. I disagree, however, that it
can be optimised and therefore solve *all* issues kdbus is trying to
address. dbus-deamon, by design, can't some things. It can't transmitt
large payloads without copying them. It can't be made race-free.

Kind regards,
-- 
Łukasz Stelmach
Samsung R Institute Poland
Samsung Electronics


signature.asc
Description: PGP signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Łukasz Stelmach]

It was 2015-04-29 śro 17:21, when Austin S Hemmelgarn wrote:
 On 2015-04-29 11:03, Theodore Ts'o wrote:
 On Wed, Apr 29, 2015 at 04:53:53PM +0200, Harald Hoyer wrote:
 Sure, I can write one binary to rule them all, pull out all the code from 
 all
 tools I need, but for me an IPC mechanism sounds a lot better. And it 
 should be
 _one_ common IPC mechanism and not a plethora of them. It should feel like 
 an
 operating system and not like a bunch of thrown together software, which is
 glued together with some magic shell scripts.

 And so requiring wireshark (and X?) in initramfs to debug problems
 once dbus is introduced is better?

 I would think shell scripts are *easier* to debug when things go
 wrong,
[...]
 I keep hearing from people that shell scripting is hard, it really
 isn't compared to a number of other scripting languages, you just need
 to actually learn to do it right (which is getting more and more
 difficult these days cause fewer and fewer CS schools are teaching
 Unix).

My 2/100 of a currency of your choice.

As much as I like(ed) shell scripts as a boot up tool and disliked
obscure boot-up procedures of some operating system, I can't help but
notice that GNU/Linux distributions have become very
sophisticated/complcated (cross out if not applicable). Personally
I feel that this degree of coplexity can't be supported by shell scripts
piping data around. It does not scale. I am not 100% sure a new IPC is
the answer, simply because I do not have experience to be so. It
definitely can be and the problem, as I see it, is real.

(The alternative answer is PowerShells capability to pipe objects. I
don't like it and I thik it's not a full answer.)

Regardless, of initrd issues I feel there is a need of a local IPC that
is more capable than UDS. Linus Torvalds is probably right that
dbus-daemon is everything but effictient. I disagree, however, that it
can be optimised and therefore solve *all* issues kdbus is trying to
address. dbus-deamon, by design, can't some things. It can't transmitt
large payloads without copying them. It can't be made race-free.

Kind regards,
-- 
Łukasz Stelmach
Samsung RD Institute Poland
Samsung Electronics


signature.asc
Description: PGP signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Richard Weinberger]

Am 30.04.2015 um 11:05 schrieb Łukasz Stelmach:
 Regardless, of initrd issues I feel there is a need of a local IPC that
 is more capable than UDS. Linus Torvalds is probably right that
 dbus-daemon is everything but effictient. I disagree, however, that it
 can be optimised and therefore solve *all* issues kdbus is trying to
 address. dbus-deamon, by design, can't some things. It can't transmitt
 large payloads without copying them. It can't be made race-free.

This is true.
But as long dbus-deamon is not optimized as much as possible there is no reason
to force push kdbus.
As soon dbus-deamon exploits all kernel interfaces as much it can and it still 
needs
work (may it performance or other stuff) we can think of new kernel features 
which
can help dbus-deamon.

Thanks,
//richard



signature.asc
Description: OpenPGP digital signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Richard Weinberger]

Am 30.04.2015 um 12:19 schrieb Łukasz Stelmach:
 It was 2015-04-30 czw 11:12, when Richard Weinberger wrote:
 Am 30.04.2015 um 11:05 schrieb Łukasz Stelmach:
 Regardless, of initrd issues I feel there is a need of a local IPC
 that is more capable than UDS. Linus Torvalds is probably right that
 dbus-daemon is everything but effictient. I disagree, however, that
 it can be optimised and therefore solve *all* issues kdbus is trying
 to address. dbus-deamon, by design, can't some things. It can't
 transmitt large payloads without copying them. It can't be made
 race-free.

 This is true.
 But as long dbus-deamon is not optimized as much as possible there is
 no reason to force push kdbus.
 As soon dbus-deamon exploits all kernel interfaces as much it can and
 it still needs work (may it performance or other stuff) we can think
 of new kernel features which can help dbus-deamon.
 
 I may not be well informed about kernel interfaces, but there are some
 use cases no dbus-daemon optimisation can make work properly because of
 rece-conditons introduced by the user-space based message router.
 
 For example, a service can't aquire credentials of a client process that
 actually sent a request (it can, but it can't trust them). The service
 can't be protected by LSM on a bus that is driven by dbus-daemon. Yes,
 dbus-daemon, can check client's and srevice's labels and enforce a
 policy but it is going to be the daemon and not the LSM code in the
 kernel.

That's why I said we can think of new kernel features if they are needed.
But they current sink or swim approach of kdbus folks is also not the solution.
As I said, if dbus-daemon utilizes the kernel interface as much as possible we
can think of new features.

Thanks,
//richard



signature.asc
Description: OpenPGP digital signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Łukasz Stelmach]

It was 2015-04-30 czw 11:12, when Richard Weinberger wrote:
 Am 30.04.2015 um 11:05 schrieb Łukasz Stelmach:
 Regardless, of initrd issues I feel there is a need of a local IPC
 that is more capable than UDS. Linus Torvalds is probably right that
 dbus-daemon is everything but effictient. I disagree, however, that
 it can be optimised and therefore solve *all* issues kdbus is trying
 to address. dbus-deamon, by design, can't some things. It can't
 transmitt large payloads without copying them. It can't be made
 race-free.

 This is true.
 But as long dbus-deamon is not optimized as much as possible there is
 no reason to force push kdbus.
 As soon dbus-deamon exploits all kernel interfaces as much it can and
 it still needs work (may it performance or other stuff) we can think
 of new kernel features which can help dbus-deamon.

I may not be well informed about kernel interfaces, but there are some
use cases no dbus-daemon optimisation can make work properly because of
rece-conditons introduced by the user-space based message router.

For example, a service can't aquire credentials of a client process that
actually sent a request (it can, but it can't trust them). The service
can't be protected by LSM on a bus that is driven by dbus-daemon. Yes,
dbus-daemon, can check client's and srevice's labels and enforce a
policy but it is going to be the daemon and not the LSM code in the
kernel.

-- 
Łukasz Stelmach
Samsung RD Institute Poland
Samsung Electronics


signature.asc
Description: PGP signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Łukasz Stelmach]

It was 2015-04-30 czw 12:40, when Richard Weinberger wrote:
 Am 30.04.2015 um 12:19 schrieb Łukasz Stelmach:
 It was 2015-04-30 czw 11:12, when Richard Weinberger wrote:
 Am 30.04.2015 um 11:05 schrieb Łukasz Stelmach:
 Regardless, of initrd issues I feel there is a need of a local IPC
 that is more capable than UDS. Linus Torvalds is probably right that
 dbus-daemon is everything but effictient. I disagree, however, that
 it can be optimised and therefore solve *all* issues kdbus is trying
 to address. dbus-deamon, by design, can't some things. It can't
 transmitt large payloads without copying them. It can't be made
 race-free.

 This is true.
 But as long dbus-deamon is not optimized as much as possible there is
 no reason to force push kdbus.
 As soon dbus-deamon exploits all kernel interfaces as much it can and
 it still needs work (may it performance or other stuff) we can think
 of new kernel features which can help dbus-deamon.
 
 I may not be well informed about kernel interfaces, but there are
 some use cases no dbus-daemon optimisation can make work properly
 because of rece-conditons introduced by the user-space based message
 router.
 
 For example, a service can't aquire credentials of a client process that
 actually sent a request (it can, but it can't trust them). The service
 can't be protected by LSM on a bus that is driven by dbus-daemon. Yes,
 dbus-daemon, can check client's and srevice's labels and enforce a
 policy but it is going to be the daemon and not the LSM code in the
 kernel.

 That's why I said we can think of new kernel features if they are
 needed.  But they current sink or swim approach of kdbus folks is also
 not the solution.  As I said, if dbus-daemon utilizes the kernel
 interface as much as possible we can think of new features.

What kernel interfaces do you suggest to use to solve the issues
I mentioned in the second paragraph: race conditions, LSM support (for
example)?

BTW. Does anyone know how microkernel-based OSes implement sockets?
-- 
Łukasz Stelmach
Samsung RD Institute Poland
Samsung Electronics


signature.asc
Description: PGP signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Richard Weinberger]

Am 30.04.2015 um 14:40 schrieb Łukasz Stelmach:
 It was 2015-04-30 czw 14:23, when Richard Weinberger wrote:
 Am 30.04.2015 um 14:16 schrieb Łukasz Stelmach:
 It was 2015-04-30 czw 12:40, when Richard Weinberger wrote:
 Am 30.04.2015 um 12:19 schrieb Łukasz Stelmach:
 It was 2015-04-30 czw 11:12, when Richard Weinberger wrote:
 Am 30.04.2015 um 11:05 schrieb Łukasz Stelmach:
 Regardless, of initrd issues I feel there is a need of a local IPC
 that is more capable than UDS. 
 [...]
 For example, a service can't aquire credentials of a client process that
 actually sent a request (it can, but it can't trust them). The service
 can't be protected by LSM on a bus that is driven by dbus-daemon. Yes,
 dbus-daemon, can check client's and srevice's labels and enforce a
 policy but it is going to be the daemon and not the LSM code in the
 kernel.

 That's why I said we can think of new kernel features if they are
 needed.  But they current sink or swim approach of kdbus folks is also
 not the solution.  As I said, if dbus-daemon utilizes the kernel
 interface as much as possible we can think of new features.

 What kernel interfaces do you suggest to use to solve the issues
 I mentioned in the second paragraph: race conditions, LSM support (for
 example)?

 The question is whether it makes sense to collect this kind of meta data.
 I really like Andy and Alan's idea improve AF_UNIX or revive AF_BUS.
 
 Race conditions have nothing to do with metadata. Neither has LSM
 support.

Sorry, I thought you mean the races while collecting metadata in userspace...

 AF_UNIX with multicast support wouldn't be AF_UNIX anymore.
 
 AF_BUS? I haven't followed the discussion back then. Why do you think it
 is better than kdbus?

Please see https://lwn.net/Articles/641278/

Thanks,
//richard



signature.asc
Description: OpenPGP digital signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Richard Weinberger]

Am 30.04.2015 um 14:16 schrieb Łukasz Stelmach:
 It was 2015-04-30 czw 12:40, when Richard Weinberger wrote:
 Am 30.04.2015 um 12:19 schrieb Łukasz Stelmach:
 It was 2015-04-30 czw 11:12, when Richard Weinberger wrote:
 Am 30.04.2015 um 11:05 schrieb Łukasz Stelmach:
 Regardless, of initrd issues I feel there is a need of a local IPC
 that is more capable than UDS. Linus Torvalds is probably right that
 dbus-daemon is everything but effictient. I disagree, however, that
 it can be optimised and therefore solve *all* issues kdbus is trying
 to address. dbus-deamon, by design, can't some things. It can't
 transmitt large payloads without copying them. It can't be made
 race-free.

 This is true.
 But as long dbus-deamon is not optimized as much as possible there is
 no reason to force push kdbus.
 As soon dbus-deamon exploits all kernel interfaces as much it can and
 it still needs work (may it performance or other stuff) we can think
 of new kernel features which can help dbus-deamon.

 I may not be well informed about kernel interfaces, but there are
 some use cases no dbus-daemon optimisation can make work properly
 because of rece-conditons introduced by the user-space based message
 router.

 For example, a service can't aquire credentials of a client process that
 actually sent a request (it can, but it can't trust them). The service
 can't be protected by LSM on a bus that is driven by dbus-daemon. Yes,
 dbus-daemon, can check client's and srevice's labels and enforce a
 policy but it is going to be the daemon and not the LSM code in the
 kernel.

 That's why I said we can think of new kernel features if they are
 needed.  But they current sink or swim approach of kdbus folks is also
 not the solution.  As I said, if dbus-daemon utilizes the kernel
 interface as much as possible we can think of new features.
 
 What kernel interfaces do you suggest to use to solve the issues
 I mentioned in the second paragraph: race conditions, LSM support (for
 example)?

The question is whether it makes sense to collect this kind of meta data.
I really like Andy and Alan's idea improve AF_UNIX or revive AF_BUS.

Thanks,
//richard



signature.asc
Description: OpenPGP digital signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Łukasz Stelmach]

It was 2015-04-30 czw 14:23, when Richard Weinberger wrote:
 Am 30.04.2015 um 14:16 schrieb Łukasz Stelmach:
 It was 2015-04-30 czw 12:40, when Richard Weinberger wrote:
 Am 30.04.2015 um 12:19 schrieb Łukasz Stelmach:
 It was 2015-04-30 czw 11:12, when Richard Weinberger wrote:
 Am 30.04.2015 um 11:05 schrieb Łukasz Stelmach:
 Regardless, of initrd issues I feel there is a need of a local IPC
 that is more capable than UDS. 
[...]
 For example, a service can't aquire credentials of a client process that
 actually sent a request (it can, but it can't trust them). The service
 can't be protected by LSM on a bus that is driven by dbus-daemon. Yes,
 dbus-daemon, can check client's and srevice's labels and enforce a
 policy but it is going to be the daemon and not the LSM code in the
 kernel.

 That's why I said we can think of new kernel features if they are
 needed.  But they current sink or swim approach of kdbus folks is also
 not the solution.  As I said, if dbus-daemon utilizes the kernel
 interface as much as possible we can think of new features.
 
 What kernel interfaces do you suggest to use to solve the issues
 I mentioned in the second paragraph: race conditions, LSM support (for
 example)?

 The question is whether it makes sense to collect this kind of meta data.
 I really like Andy and Alan's idea improve AF_UNIX or revive AF_BUS.

Race conditions have nothing to do with metadata. Neither has LSM
support.

AF_UNIX with multicast support wouldn't be AF_UNIX anymore.

AF_BUS? I haven't followed the discussion back then. Why do you think it
is better than kdbus?

-- 
Łukasz Stelmach
Samsung RD Institute Poland
Samsung Electronics


signature.asc
Description: PGP signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Richard Weinberger]

Am 30.04.2015 um 16:52 schrieb Łukasz Stelmach:
 Sorry, I thought you mean the races while collecting metadata in userspace...
 
 My bad, some reace conditions *are* associated with collecting metadata
 but ont all. It is impossible (correct me if I am wrong) to implement
 reliable die-on-idle with dbus-daemon.

IIRC Andy gave some ideas howto deal with that.
i.e. https://lkml.org/lkml/2015/4/29/622

 AF_UNIX with multicast support wouldn't be AF_UNIX anymore.

 AF_BUS? I haven't followed the discussion back then. Why do you think it
 is better than kdbus?

 Please see https://lwn.net/Articles/641278/
 
 Thanks. If I understand correctly, the author suggests using EBPF on a
 receiveing socket side for receiving multicast messages. This is nice if
 you care about introducing (or not) (too?) much of new code. However,
 AFAICT it may be more computationally complex than Bloom filters because
 you need to run EBPF on every receiving socket instead of getting a list
 of a few of them to copy data to. Of course for small number of
 receivers the constant cost of running the Bloom filter may be higher.

To make the story short, the kdbus *concept* needs much more
thought. There are many ideas out there howto deal with dbus issues without 
introducing
an ad-hoc solution. AF_BUS is just one of them.
IMHO AF_BUS would be nice but the decision is not up to me.

Thanks,
//richard



signature.asc
Description: OpenPGP digital signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Eric W. Biederman]

On April 29, 2015 7:47:53 AM CDT, Harald Hoyer har...@redhat.com wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 29.04.2015 01:12, John Stoffel wrote:
 LDAP is pretty damn generic, in that you can put pretty large objects
into
 it, and pretty large OUs, etc.  So why would it be a candidate for
going
 into the kernel?  And why is kdbus so important in the kernel as
well?
 People have talked about it needing to be there for bootup, but isn't
that
 why we ripped out RAID detection and such from the kernel and built
 initramfs, so that there's LESS in the kernel, and more in an early
 userspace?  Same idea with dbus in my opinion.

Let me elaborate on the initramfs/shutdown situation a little bit more,
because I have to deal with that every day.

Because of the let's move everything to userspace sentiment we
nowadays
have the situation, that we need a lot of tools to setup the root
device.

Be it LVM on IMSM or iSCSI multipath, the initramfs has to setup the
network
(with bridging, bonding, etc.), the iSCSI connection, assemble the
raid, the
LVM, open crypto devices, etc...
And if something goes wrong, you want to have a shell, see all the logs
and
debug things.

Now over the time we moved away from simple shell scripts (without any
logging) and static compiled special versions for the initramfs to a
mini
distribution in the initramfs, which simplifies maintenance and
improves
reliability.

Basically you want to use the same tools in the initramfs (and
shutdown)
which you already have and use in your real root, with the same
configuration
files and the same interfaces and the same code paths.

Therefore systemd is started in dracut created initramfs, which starts
journald for logging. The same basic systemd targets exist in the
initramfs
as on the real root, so normally you don't have to cope with
specialized
versions for the initramfs.

The target here is to have the same IPC mechanism from the very
beginning to
the very end. No crappy fallback mechanisms in case a daemon is not
running
or has crashed, no creepy transition from initramfs root to real root
to
shutdown root.

We already have such transitions like: systemd, journald, mdmon [1],
etc.
systemd has to serialize itself, journald's file descriptors are
transitioned
over, mdmon jumps through hoops. Remember you want to get rid of open
files
and executables and have to reexec everything, if you transition from
the
initramfs root to the real root, and also from the real root to the
shutdown
root.

We really don't want the IPC mechanism to be in a flux state. All tools
have
to fallback to a non-standard mechanism in that case.

If I have to pull in a dbus daemon in the initramfs, we still have the
chicken and egg problem for PID 1 talking to the logging daemon and
starting
dbus.
systemd cannot talk to journald via dbus unless dbus-daemon is started,
dbus
cannot log anything on startup, if journald is not running, etc...

dbus-daemon would have to transition to the real root, and from the
real root
to the shutdown root, without losing state.

Which does not sound fundamentally hard.  
Unify the roots, and make /run or wherever the dbus socket lives always 
available.  As long as your initramfs has the latest versions of software there 
is no need for any tricky transitions except to upgrade software on a running 
system.


Of course this can all be done, but it would involve fallback
mechanisms,
which we want to get rid off. 

Only if you design things poorly.

Hopefully, you don't suggest to merge
dbus with
PID 1. Also with a daemon, you will lose the points mentioned in the
cover mail

I don't see how something that is inappropriate to be in PID 1 is better in PID 
0.


I don't care, if the kdbus speedup is only marginal.

In my ideal world, there is a standard IPC mechanism from the beginning
to
the end, which does not rely on any process running (except the kernel)
and
which is used by _all_ tools, be it a system daemon providing
information and
interfaces about device assembly or network setup tools or end user
desktop
processes.

And that is a beautiful dream and an absolutely rubbish way to get there.  If 
the performance is not top notch everything can not use your beautiful IPC 
mechanism.Which means your dream fails.

Good performance is a hard requirement to get where you want to be.

dbus _is_ such an easy, flexible standard IPC mechanism. Of course, you
can
invent the wheel again (NIH, we know better) and wait and see, if
that
works out. Until then the whole common IPC problem is unresolved and
Linux
distributions are just a collection of random software with no common
interoperability and home grown interfaces.


kdbus seems to be the NIH we know better better approach.  Many of it's 
design decisions we have chosen to differently elsewhere in the kernel because 
the have caused problems.  When these issues have been pointed out in review 
people have blown off leading to the current mess.

Furthermore I don't know that I 

Re: [GIT PULL] kdbus for 4.1-rc1

[Łukasz Stelmach]

It was 2015-04-30 czw 14:45, when Richard Weinberger wrote:
 Am 30.04.2015 um 14:40 schrieb Łukasz Stelmach:
 It was 2015-04-30 czw 14:23, when Richard Weinberger wrote:
 Am 30.04.2015 um 14:16 schrieb Łukasz Stelmach:
 It was 2015-04-30 czw 12:40, when Richard Weinberger wrote:
 Am 30.04.2015 um 12:19 schrieb Łukasz Stelmach:
 It was 2015-04-30 czw 11:12, when Richard Weinberger wrote:
 Am 30.04.2015 um 11:05 schrieb Łukasz Stelmach:
 Regardless, of initrd issues I feel there is a need of a local IPC
 that is more capable than UDS. 
 [...]
 For example, a service can't aquire credentials of a client process that
 actually sent a request (it can, but it can't trust them). The service
 can't be protected by LSM on a bus that is driven by dbus-daemon. Yes,
 dbus-daemon, can check client's and srevice's labels and enforce a
 policy but it is going to be the daemon and not the LSM code in the
 kernel.

 That's why I said we can think of new kernel features if they are
 needed.  But they current sink or swim approach of kdbus folks is also
 not the solution.  As I said, if dbus-daemon utilizes the kernel
 interface as much as possible we can think of new features.

 What kernel interfaces do you suggest to use to solve the issues
 I mentioned in the second paragraph: race conditions, LSM support (for
 example)?

 The question is whether it makes sense to collect this kind of meta data.
 I really like Andy and Alan's idea improve AF_UNIX or revive AF_BUS.
 
 Race conditions have nothing to do with metadata. Neither has LSM
 support.

 Sorry, I thought you mean the races while collecting metadata in userspace...

My bad, some reace conditions *are* associated with collecting metadata
but ont all. It is impossible (correct me if I am wrong) to implement
reliable die-on-idle with dbus-daemon.

 AF_UNIX with multicast support wouldn't be AF_UNIX anymore.
 
 AF_BUS? I haven't followed the discussion back then. Why do you think it
 is better than kdbus?

 Please see https://lwn.net/Articles/641278/

Thanks. If I understand correctly, the author suggests using EBPF on a
receiveing socket side for receiving multicast messages. This is nice if
you care about introducing (or not) (too?) much of new code. However,
AFAICT it may be more computationally complex than Bloom filters because
you need to run EBPF on every receiving socket instead of getting a list
of a few of them to copy data to. Of course for small number of
receivers the constant cost of running the Bloom filter may be higher.

Kind regards,
-- 
Łukasz Stelmach
Samsung RD Institute Poland
Samsung Electronics


signature.asc
Description: PGP signature

Re: [GIT PULL] kdbus for 4.1-rc1

[John Stoffel]

> "David" == David Herrmann  writes:

David> Hi
David> On Wed, Apr 29, 2015 at 10:43 PM, David Lang  wrote:
>> If the justification for why this needs to be in the kernel is that you
>> can't reliably prevent apps from exiting if there are pending messages, [...]

David> It's not.

>> the answer of "preventing apps from exiting if there are pending messages
>> isn't a sane thing to try and do" is a direct counter to that justification
>> for including it in the kernel.

David> It's optionally used for reliable exit-on-idle.

Then why is there a critical race that must be solved in the kernel if
it's optional?  And can you please describe in more detail what this
'exit-on-idle' thing is and how it works and why you would use it?


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Dave Airlie]

>>>
>>> I've had Enterprise systems where I could hit power on two boxes, and
>>> finish
>>> the OS install on one before the other has even finished POST and look
>>> for
>>> the boot media. I did this 5 years ago, before the "let's speed up boot"
>>> push started.
>>>
>>> Admittedly, this wasn't a stock distro boot/install, it was my own
>>> optimized
>>> one, but it also wasn't as optimized and automated as it could have been
>>> (several points where the installer needed to pick items from a menu and
>>> enter values)
>>>
>>
>> You guys might have missed this new industry trend, I think they call
>> it virtualisation,
>>
>> I hear it's going to be big, you might want to look into it.
>
>
> So what do you run your virtual machines on? you still have to put an OS on
> the hardware to support your VMs. Virtualization doesn't eliminate servers
> (as much as some cloud advocates like to claim it does)
>
> And virtualization has overhead, sometimes very significant overhead, so
> it's not always the right answer.
>

Thanks for proving my point, RHEL as a distro runs in both scenarios, optimising
one is important at the moment, that fact that it might speed up boot on server
which take 15 mins to POST is a side effect.

For some reason people seem to think their one use case is all that matters,

Dave.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[David Lang]

On Thu, 30 Apr 2015, Dave Airlie wrote:


On 30 April 2015 at 10:05, David Lang  wrote:

On Wed, 29 Apr 2015, Theodore Ts'o wrote:


On Wed, Apr 29, 2015 at 12:26:59PM -0400, John Stoffel wrote:


If your customers wnat this feature, you're more than welcome to fork
the kernel and support it yourself.  Oh wait... Redhat does that
already.  So what's the problem?   Just put it into RHEL (which I use
I admit, along with Debian/Mint) and be done with it.



Harald,

If you make the RHEL initramfs harder to debug in the field, I will
await the time when some Red Hat field engineers will need to do the
same sort of thing I have had to do in the field, and be amused when
they want to shake you very warmly by the throat.  :-)

Seriously, keep things as simple as possible in the initramfs; don't
use complicated bus protocols; that way lies madness.  Enterprise
systems aren't constantly booting (or they shouldn't be, if your
kernels are sufficiently reliable :-), so trying to optimize for an
extra 2 or 3 seconds worth of boot time really, REALLY isn't worth it.



I've had Enterprise systems where I could hit power on two boxes, and finish
the OS install on one before the other has even finished POST and look for
the boot media. I did this 5 years ago, before the "let's speed up boot"
push started.

Admittedly, this wasn't a stock distro boot/install, it was my own optimized
one, but it also wasn't as optimized and automated as it could have been
(several points where the installer needed to pick items from a menu and
enter values)



You guys might have missed this new industry trend, I think they call
it virtualisation,

I hear it's going to be big, you might want to look into it.


So what do you run your virtual machines on? you still have to put an OS on the 
hardware to support your VMs. Virtualization doesn't eliminate servers (as much 
as some cloud advocates like to claim it does)


And virtualization has overhead, sometimes very significant overhead, so it's 
not always the right answer.


David Lang
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Dave Airlie]

On 30 April 2015 at 10:05, David Lang  wrote:
> On Wed, 29 Apr 2015, Theodore Ts'o wrote:
>
>> On Wed, Apr 29, 2015 at 12:26:59PM -0400, John Stoffel wrote:
>>>
>>> If your customers wnat this feature, you're more than welcome to fork
>>> the kernel and support it yourself.  Oh wait... Redhat does that
>>> already.  So what's the problem?   Just put it into RHEL (which I use
>>> I admit, along with Debian/Mint) and be done with it.
>>
>>
>> Harald,
>>
>> If you make the RHEL initramfs harder to debug in the field, I will
>> await the time when some Red Hat field engineers will need to do the
>> same sort of thing I have had to do in the field, and be amused when
>> they want to shake you very warmly by the throat.  :-)
>>
>> Seriously, keep things as simple as possible in the initramfs; don't
>> use complicated bus protocols; that way lies madness.  Enterprise
>> systems aren't constantly booting (or they shouldn't be, if your
>> kernels are sufficiently reliable :-), so trying to optimize for an
>> extra 2 or 3 seconds worth of boot time really, REALLY isn't worth it.
>
>
> I've had Enterprise systems where I could hit power on two boxes, and finish
> the OS install on one before the other has even finished POST and look for
> the boot media. I did this 5 years ago, before the "let's speed up boot"
> push started.
>
> Admittedly, this wasn't a stock distro boot/install, it was my own optimized
> one, but it also wasn't as optimized and automated as it could have been
> (several points where the installer needed to pick items from a menu and
> enter values)
>

You guys might have missed this new industry trend, I think they call
it virtualisation,

I hear it's going to be big, you might want to look into it.

Dave.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[David Lang]

On Wed, 29 Apr 2015, Theodore Ts'o wrote:


On Wed, Apr 29, 2015 at 12:26:59PM -0400, John Stoffel wrote:

If your customers wnat this feature, you're more than welcome to fork
the kernel and support it yourself.  Oh wait... Redhat does that
already.  So what's the problem?   Just put it into RHEL (which I use
I admit, along with Debian/Mint) and be done with it.


Harald,

If you make the RHEL initramfs harder to debug in the field, I will
await the time when some Red Hat field engineers will need to do the
same sort of thing I have had to do in the field, and be amused when
they want to shake you very warmly by the throat.  :-)

Seriously, keep things as simple as possible in the initramfs; don't
use complicated bus protocols; that way lies madness.  Enterprise
systems aren't constantly booting (or they shouldn't be, if your
kernels are sufficiently reliable :-), so trying to optimize for an
extra 2 or 3 seconds worth of boot time really, REALLY isn't worth it.


I've had Enterprise systems where I could hit power on two boxes, and finish the 
OS install on one before the other has even finished POST and look for the boot 
media. I did this 5 years ago, before the "let's speed up boot" push started.


Admittedly, this wasn't a stock distro boot/install, it was my own optimized 
one, but it also wasn't as optimized and automated as it could have been 
(several points where the installer needed to pick items from a menu and enter 
values)


David Lang
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Theodore Ts'o]

On Wed, Apr 29, 2015 at 12:26:59PM -0400, John Stoffel wrote:
> If your customers wnat this feature, you're more than welcome to fork
> the kernel and support it yourself.  Oh wait... Redhat does that
> already.  So what's the problem?   Just put it into RHEL (which I use
> I admit, along with Debian/Mint) and be done with it.

Harald,

If you make the RHEL initramfs harder to debug in the field, I will
await the time when some Red Hat field engineers will need to do the
same sort of thing I have had to do in the field, and be amused when
they want to shake you very warmly by the throat.  :-)

Seriously, keep things as simple as possible in the initramfs; don't
use complicated bus protocols; that way lies madness.  Enterprise
systems aren't constantly booting (or they shouldn't be, if your
kernels are sufficiently reliable :-), so trying to optimize for an
extra 2 or 3 seconds worth of boot time really, REALLY isn't worth it.

  - Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[John Stoffel]

> "Austin" == Austin S Hemmelgarn  writes:

Austin> On 2015-04-29 14:54, Andy Lutomirski wrote:
>> On Apr 29, 2015 5:48 AM, "Harald Hoyer"  wrote:
>>> 
>>> * Being in the kernel closes a lot of races which can't be fixed with
>>> the current userspace solutions.  For example, with kdbus, there is a
>>> way a client can disconnect from a bus, but do so only if no further
>>> messages present in its queue, which is crucial for implementing
>>> race-free "exit-on-idle" services
>> 
>> This can be implemented in userspace.
>> 
>> Client to dbus daemon: may I exit now?
>> Dbus daemon to client: yes (and no more messages) or no
>> 

Austin> Depending on how this is implemented, there would be a
Austin> potential issue if a message arrived for the client after the
Austin> daemon told it it could exit, but before it finished shutdown,
Austin> in which case the message might get lost.

What makes anyone think they can guarrantee that a message is even
received?  I could see the daemon sending the message and the client
getting a segfault and dumping core.  What then?  How would kdbus
solve this type of "race" anyway? 

Can anyone give a concrete example of one of the races that are closed
here?  That's been one of the missing examples.  And remember, there's
no perfection.  Even in the kernel we just had a discussion about
missed/missing IPIs and lost processor interrupts, etc.  Expecting
perfection is just asking for trouble.  

That's why there are timeouts, retries and just giving up and throwing
an exception.  

John
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Paul Moore]

On Fri, Apr 24, 2015 at 4:08 AM, Karol Lewandowski
 wrote:
> On Thu, Apr 23, 2015 at 09:30:13PM +0200, Greg Kroah-Hartman wrote:
>> On Thu, Apr 23, 2015 at 01:42:25PM -0400, Stephen Smalley wrote:
>> > On 04/23/2015 01:16 PM, Greg Kroah-Hartman wrote:
>> > > The binder developers at Samsung have stated that the implementation we
>> > > have here works for their model as well, so I guess that is some kind of
>> > > verification it's not entirely tied to D-Bus.  They have plans on
>> > > dropping the existing binder kernel code and using the kdbus code
>> > > instead when it is merged.
>> >
>> > Where do things stand wrt LSM hooks for kdbus?  I don't see any security
>> > hook calls in the kdbus tree except for the purpose of metadata
>> > collection of process security labels.  But nothing for enforcing MAC
>> > over kdbus IPC.  binder has a set of security hooks for that purpose, so
>> > it would be a regression wrt MAC enforcement to switch from binder to
>> > kdbus without equivalent checking there.
>>
>> There was a set of LSM hooks proposed for kdbus posted by Karol
>> Lewandowsk last October, and it also included SELinux and Smack patches.
>> They were going to be refreshed based on the latest code changes, but I
>> haven't seen them posted, or I can't seem to find them in my limited
>> email archive.
>
> We have been waiting for right moment with these. :-)
>
>> Karol, what's the status of them?
>
> I have handed patchset over to Paul Osmialowski who started rework it for v4
> relatively recently.  I think it shouldn't be that hard to post updated 
> version...
>
> Paul?

Different Paul here, but very interested in the LSM and SELinux hooks
for obvious reasons; at a bare minimum please CC the LSM list on the
kdbus hooks, and preferably the SELinux list as well.  The initial
SELinux hooks I threw together were just a rough first pass, we (the
LSM and SELinux folks) need to have a better discussion about how to
provide the necessary access controls for kdbus ... preferably before
it finds its way into a released kernel.

-- 
paul moore
www.paul-moore.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[David Herrmann]

Hi

On Wed, Apr 29, 2015 at 10:43 PM, David Lang  wrote:
> If the justification for why this needs to be in the kernel is that you
> can't reliably prevent apps from exiting if there are pending messages, [...]

It's not.

> the answer of "preventing apps from exiting if there are pending messages
> isn't a sane thing to try and do" is a direct counter to that justification
> for including it in the kernel.

It's optionally used for reliable exit-on-idle.

Thanks
David
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[David Lang]

On Wed, 29 Apr 2015, Andy Lutomirski wrote:


On Wed, Apr 29, 2015 at 1:15 PM, David Lang  wrote:

On Wed, 29 Apr 2015, Andy Lutomirski wrote:


On Wed, Apr 29, 2015 at 12:30 PM, Austin S Hemmelgarn
 wrote:


On 2015-04-29 14:54, Andy Lutomirski wrote:



On Apr 29, 2015 5:48 AM, "Harald Hoyer"  wrote:




  * Being in the kernel closes a lot of races which can't be fixed with
the current userspace solutions.  For example, with kdbus, there is
a
way a client can disconnect from a bus, but do so only if no
further
messages present in its queue, which is crucial for implementing
race-free "exit-on-idle" services




This can be implemented in userspace.

Client to dbus daemon: may I exit now?
Dbus daemon to client: yes (and no more messages) or no


Depending on how this is implemented, there would be a potential issue if
a
message arrived for the client after the daemon told it it could exit,
but
before it finished shutdown, in which case the message might get lost.



Then implement it the right way?  The client sends some kind of
sequence number with its request.



so any app in the system can prevent any other app from exiting/restarting
by just sending it the equivalent of a ping over dbus?

preventing an app from exiting because there are unhandled messages doesn't
mean that those messages are going to be handled, just that they will get
read and dropped on the floor by an app trying to exit. Sometimes you will
just end up with a hung app that can't process messages and needs to be
restarted, but can't be restarted because there are pending messages.


I think this consideration is more or less the same whether it's
handled in the kernel or in userspace, though.


If the justification for why this needs to be in the kernel is that you can't 
reliably prevent apps from exiting if there are pending messages, then the 
answer of "preventing apps from exiting if there are pending messages isn't a 
sane thing to try and do" is a direct counter to that justification for 
including it in the kernel.


David Lang
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Andy Lutomirski]

On Wed, Apr 29, 2015 at 1:15 PM, David Lang  wrote:
> On Wed, 29 Apr 2015, Andy Lutomirski wrote:
>
>> On Wed, Apr 29, 2015 at 12:30 PM, Austin S Hemmelgarn
>>  wrote:
>>>
>>> On 2015-04-29 14:54, Andy Lutomirski wrote:


 On Apr 29, 2015 5:48 AM, "Harald Hoyer"  wrote:
>
>
>
>   * Being in the kernel closes a lot of races which can't be fixed with
> the current userspace solutions.  For example, with kdbus, there is
> a
> way a client can disconnect from a bus, but do so only if no
> further
> messages present in its queue, which is crucial for implementing
> race-free "exit-on-idle" services



 This can be implemented in userspace.

 Client to dbus daemon: may I exit now?
 Dbus daemon to client: yes (and no more messages) or no

>>> Depending on how this is implemented, there would be a potential issue if
>>> a
>>> message arrived for the client after the daemon told it it could exit,
>>> but
>>> before it finished shutdown, in which case the message might get lost.
>>>
>>
>> Then implement it the right way?  The client sends some kind of
>> sequence number with its request.
>
>
> so any app in the system can prevent any other app from exiting/restarting
> by just sending it the equivalent of a ping over dbus?
>
> preventing an app from exiting because there are unhandled messages doesn't
> mean that those messages are going to be handled, just that they will get
> read and dropped on the floor by an app trying to exit. Sometimes you will
> just end up with a hung app that can't process messages and needs to be
> restarted, but can't be restarted because there are pending messages.

I think this consideration is more or less the same whether it's
handled in the kernel or in userspace, though.

--Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[David Lang]

On Wed, 29 Apr 2015, Andy Lutomirski wrote:


On Wed, Apr 29, 2015 at 12:30 PM, Austin S Hemmelgarn
 wrote:

On 2015-04-29 14:54, Andy Lutomirski wrote:


On Apr 29, 2015 5:48 AM, "Harald Hoyer"  wrote:



  * Being in the kernel closes a lot of races which can't be fixed with
the current userspace solutions.  For example, with kdbus, there is a
way a client can disconnect from a bus, but do so only if no further
messages present in its queue, which is crucial for implementing
race-free "exit-on-idle" services



This can be implemented in userspace.

Client to dbus daemon: may I exit now?
Dbus daemon to client: yes (and no more messages) or no


Depending on how this is implemented, there would be a potential issue if a
message arrived for the client after the daemon told it it could exit, but
before it finished shutdown, in which case the message might get lost.



Then implement it the right way?  The client sends some kind of
sequence number with its request.


so any app in the system can prevent any other app from exiting/restarting by 
just sending it the equivalent of a ping over dbus?


preventing an app from exiting because there are unhandled messages doesn't mean 
that those messages are going to be handled, just that they will get read and 
dropped on the floor by an app trying to exit. Sometimes you will just end up 
with a hung app that can't process messages and needs to be restarted, but can't 
be restarted because there are pending messages.


The problem with "guaranteed delivery" messages is that things _will_ go wrong 
that will cause the messages to not be received and processed. At that point you 
have the choice of loosing some messages or freezing your entire system (you can 
buffer them for some time, but eventually you will run out of buffer space)


We see this all the time in the logging world, people configure their systems 
for reliable delivery of log messages to a remote machine, then when that remote 
machine goes down and can't receive messages (or a network issue blocks the 
traffic), the sending machine blocks and causes an outage.


Being too strict about guaranteeing delivery just doesn't work. You must have a 
mechanism to abort and throw away unprocessed messages. If this means 
disconnecting the receiver so that there are no missing messages to the 
receiver, that's a valid choice. But preventing a receiver from exiting because 
it hasn't processed a message is not a valid choice.


David Lang
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Andy Lutomirski]

On Wed, Apr 29, 2015 at 12:30 PM, Austin S Hemmelgarn
 wrote:
> On 2015-04-29 14:54, Andy Lutomirski wrote:
>>
>> On Apr 29, 2015 5:48 AM, "Harald Hoyer"  wrote:
>>>
>>>
>>>   * Being in the kernel closes a lot of races which can't be fixed with
>>> the current userspace solutions.  For example, with kdbus, there is a
>>> way a client can disconnect from a bus, but do so only if no further
>>> messages present in its queue, which is crucial for implementing
>>> race-free "exit-on-idle" services
>>
>>
>> This can be implemented in userspace.
>>
>> Client to dbus daemon: may I exit now?
>> Dbus daemon to client: yes (and no more messages) or no
>>
> Depending on how this is implemented, there would be a potential issue if a
> message arrived for the client after the daemon told it it could exit, but
> before it finished shutdown, in which case the message might get lost.
>

Then implement it the right way?  The client sends some kind of
sequence number with its request.

--Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Austin S Hemmelgarn]

On 2015-04-29 14:54, Andy Lutomirski wrote:

On Apr 29, 2015 5:48 AM, "Harald Hoyer"  wrote:


  * Being in the kernel closes a lot of races which can't be fixed with
the current userspace solutions.  For example, with kdbus, there is a
way a client can disconnect from a bus, but do so only if no further
messages present in its queue, which is crucial for implementing
race-free "exit-on-idle" services


This can be implemented in userspace.

Client to dbus daemon: may I exit now?
Dbus daemon to client: yes (and no more messages) or no

Depending on how this is implemented, there would be a potential issue 
if a message arrived for the client after the daemon told it it could 
exit, but before it finished shutdown, in which case the message might 
get lost.




smime.p7s
Description: S/MIME Cryptographic Signature

Re: [GIT PULL] kdbus for 4.1-rc1

[John Stoffel]

> "Steven" == Steven Rostedt  writes:

Steven> On Wed, Apr 29, 2015 at 12:26:59PM -0400, John Stoffel wrote:
>> 
>> If your customers wnat this feature, you're more than welcome to fork
>> the kernel and support it yourself.  Oh wait... Redhat does that
>> already.  So what's the problem?   Just put it into RHEL (which I use
>> I admit, along with Debian/Mint) and be done with it.

Steven> Red Hat tries very hard to push things upstream. It's policy
Steven> is to not keep things for themselves, but always work with the
Steven> community. That way, everyone benefits. Ideally, we should
Steven> come up with a solution that works for all.

Yeah, I agree they have been good.  I'm just reacting to the off the
cuff comment of "my customers need it" which isn't a justification for
this feature, esp when it hasn't been shown to be needed in the
kernel.  

We went through alot of this with tux the in-kernel httpd server, and
pushing other stuff out to user-space over the years.  Why this needs
to come in isn't clear.  Or why not just a small part needing to come
in with the rest in userspace.   

John
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Martin Steigerwald]

Am Mittwoch, 29. April 2015, 13:39:42 schrieb Steven Rostedt:
> On Wed, Apr 29, 2015 at 12:26:59PM -0400, John Stoffel wrote:
> > If your customers wnat this feature, you're more than welcome to fork
> > the kernel and support it yourself.  Oh wait... Redhat does that
> > already.  So what's the problem?   Just put it into RHEL (which I use
> > I admit, along with Debian/Mint) and be done with it.
> 
> Red Hat tries very hard to push things upstream. It's policy is to not
> keep things for themselves, but always work with the community. That
> way, everyone benefits. Ideally, we should come up with a solution that
> works for all.

I think work with the community is a two-way process.

Two way as in actually really *listening* to feedback instead of trying to 
push things as much as possible, believing to be *right* about things. I 
honestly dislike the "I know it better than you, go away" kind of attitude 
I have seen again and again. Here, in systemd-devel (where I unsubscribed 
again as I saw no use in continuing the discussion there) and even in 
Debian mailinglists and bug reports.

Wherever I look what I call the "systemd" approach triggers intense 
polarity and resistance. With that I do not say there is something wrong 
about it, yet, I ask myself, why is that? And my best answer I came up 
with up to now comes back to how proponents of the new, different, not 
necessary better or worse, way, treat feedback.

There I found to some extent: taking the feedback into account and 
actually adressing it. Especially when the feedback fitted into the new way 
of doing things.

Yet I also found:

- "I know it better than you, go away."

- "Please only stick to pure technical reasons" as in "Whats wrong with 
the code?" disregarding any concerns about the *concept* and about 
different oppinions about whether kdbus code actually really belongs into 
the kernel

- Ignoring it


So I still say the issues are not purely technical. So I think a purely 
technical as in "what´s wrong with the code?" approach will not address 
the core of this discussion and the strong resistance against merging 
kdbus into the kernel.

It sometimes appears to me like childs arguing about whether to paint 
their favorite toy red or green. I think a healthy approach might be to 
agree to disagree and work from there. That would at least break the "I am 
right", "No, I am right" cycle.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Andy Lutomirski]

On Apr 29, 2015 5:48 AM, "Harald Hoyer"  wrote:
> Of course this can all be done, but it would involve fallback mechanisms,
> which we want to get rid off. Hopefully, you don't suggest to merge dbus with
> PID 1. Also with a daemon, you will lose the points mentioned in the cover 
> mail
> :
>
>  * Security: The peers which communicate do not have to trust each
>other, as the only trustworthy component in the game is the kernel
>which adds metadata and ensures that all data passed as payload is
>either copied or sealed, so that the receiver can parse the data
>without having to protect against changing memory while parsing
>buffers. Also, all the data transfer is controlled by the kernel,
>so that LSMs can track and control what is going on, without
>involving userspace. Because of the LSM issue, security people are
>much happier with this model than the current scheme of having to
>hook into dbus to mediate things.

Other security people prefer code to stay out of the kernel when possible.

Also, this metadata argument is still invalid.  Sockets can be
improved for this purpose.

>
>  * Being in the kernel closes a lot of races which can't be fixed with
>the current userspace solutions.  For example, with kdbus, there is a
>way a client can disconnect from a bus, but do so only if no further
>messages present in its queue, which is crucial for implementing
>race-free "exit-on-idle" services

This can be implemented in userspace.

Client to dbus daemon: may I exit now?
Dbus daemon to client: yes (and no more messages) or no

>
>  * Eavesdropping on the kernel level, so privileged users can hook into
>the message stream without hacking support for that into their
>userspace processes
>

Why would it be a hack in userspace but not a hack in the kernel?

>  * A number of smaller benefits: for example kdbus learned a way to peek
>full messages without dequeing them, which is really useful for
>logging metadata when handling bus-activation requests.

MSG_PEEK?

>
> I don't care, if the kdbus speedup is only marginal.
>
> In my ideal world, there is a standard IPC mechanism from the beginning to
> the end, which does not rely on any process running (except the kernel) and
> which is used by _all_ tools, be it a system daemon providing information and
> interfaces about device assembly or network setup tools or end user desktop
> processes.
>
> dbus _is_ such an easy, flexible standard IPC mechanism. Of course, you can
> invent the wheel again (NIH, "we know better") and wait and see, if that
> works out. Until then the whole common IPC problem is unresolved and Linux
> distributions are just a collection of random software with no common
> interoperability and home grown interfaces.

I don't think anyone is suggesting throwing out dbus.

>
> [1] transitioning mdmon is one of the critical parts for an IMSM raid array.
> Also running an executable from a disk, which the executable is monitoring,
> and which stops functioning, if the executable is not responding is insane.
>

That sounds like an excellent reason to just keep running the
initramfs copy, and I don't see why IPC has anything to do with where
the executable comes from.

--Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Martin Steigerwald]

Am Mittwoch, 29. April 2015, 17:22:08 schrieb Harald Hoyer:
> On 29.04.2015 17:17, Austin S Hemmelgarn wrote:
> > On 2015-04-29 11:07, Harald Hoyer wrote:
> >> Most of the stuff does not work without udev and something like
> >> systemd.> 
> > That's funny, apparently the initramfs images I've been using for
> > multiple months now on server systems at work which don't have
> > systemd, udev, or dbus, and do LVM/RAID assembly, network
> > configuration, crypto devices, multipath, many different filesystems,
> > and a number of other oddball configurations due to the insanity that
> > is the software I have to deal with from our company, don't work.  I
> > wonder how my systems are booting successfully 100% of the time then?
> Then you should probably open source your initramfs, so we can all
> benefit from it and use it for all distributions.

Do you really think that the tooling will make that much of a difference?

I think there will always be cases where a initramfs will not work until 
adapted to it. And then its nice, to be able to do things like this:

merkaba:~> cat /etc/initramfs-tools/scripts/local-top/btrfs
#!/bin/sh

PREREQ="lvm"
prereqs()
{
echo $PREREQ
}

case $1 in
prereqs)
prereqs
exit 0;
esac

. /scripts/functions

log_begin_msg "Initializing BTRFS RAID-1."

modprobe btrfs
vgchange -ay
btrfs device scan

log_end_msg


How would I add support for some configuration that a systemd or purely 
dracut + udev based initramfs does not support *yet*, on my own?

Yes, one can argue, why doesn´t Debian support it already, but heck, I can 
do it myself and report a bug about it, without having to fire up a C 
compiler in order to fix things. I may be able to do this myself, but at a 
much higher cost in time.

Above thing works so long already that I even often forgot about it.

That said, if I still get a chance to execute a script at some time, a 
dracut based initramfs may just be totally fine with it, but I want this 
possibility and a shell to fix things up myself it they go wrong. And while 
I do not get the need for having systemd in the initramfs at all, I might 
be fine with it, if I can fix things up myself in case of problems.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Stephen Smalley]

On 04/29/2015 11:18 AM, Simon McVittie wrote:
> On 29/04/15 14:35, Stephen Smalley wrote:
>> It is also interesting that kdbus allows impersonation of any
>> credential, including security label, by "privileged" clients, where
>> privileged simply means it either has CAP_IPC_OWNER or owns (euid
>> matches uid) the bus.
> 
> FWIW, this particular feature is *not* one of those that are necessary
> for feature parity with dbus-daemon. There's no API for making
> dbus-daemon fake its clients' credentials; if you can ptrace it, then
> you can of course subvert it arbitrarily, but nothing less hackish than
> that is currently offered.

Then I'd be inclined to drop it from kdbus unless some compelling use
case exists, and even then, I don't believe that CAP_IPC_OWNER or
bus-owner uid match is sufficient even for forging credentials other
than the security label.  For socket credentials passing, for example,
the kernel checks CAP_SYS_ADMIN for pid forging, CAP_SETUID for uid
forging, and CAP_SETGID for gid forging.  And I don't believe we support
any form of forging of the security label on socket credentials.

> For feature parity with dbus-daemon, the fact that
> eavesdropping/monitoring *exists* is necessary (it's a widely used
> developer/sysadmin feature) but the precise mechanics of how you get it
> are not necessarily set in stone. In particular, if you think kdbus'
> definition of "are you privileged?" may be too broad, that seems a valid
> question to be asking.
> 
> In traditional D-Bus, individual users can normally eavesdrop/monitor on
> their own session buses (which are not a security boundary, unless
> specially reconfigured), and this is a useful property; on non-LSM
> systems without special configuration, each user should ideally be able
> to monitor their own kdbus user bus, too.
> 
> The system bus *is* a security boundary, and administrative privileges
> should be required to eavesdrop on it. At a high level, someone with
> "full root privileges" should be able to eavesdrop, and ordinary users
> should not; there are various possible criteria for distinguishing
> between those two extremes, and I have no opinion on whether
> CAP_IPC_OWNER is the most appropriate cutoff point.
> 
> In dbus-daemon, LSMs with integration code in dbus-daemon have the
> opportunity to mediate eavesdropping specially. SELinux does not
> currently do this (as far as I can see), but AppArmor does, so
> AppArmor-confined processes are not normally allowed to eavesdrop on the
> session bus (even though the same user's unconfined processes may). That
> seems like one of the obvious places for an LSM hook in kdbus.

Yes, we would want to control this in SELinux; I suspect that either the
eavesdropping functionality did not exist in dbus-daemon at the time of
the original dbus-daemon SELinux integration or it was an oversight.

> Having eavesdropping be unobservable means that applications cannot
> change their behaviour while they are being watched, either maliciously
> (to hide from investigation) or accidentally (bugs that only happen when
> not being debugged are the hardest to fix). dbus-daemon's traditional
> implementation of eavesdropping has had side-effects in the past, which
> is undesirable, and is addressed by the new monitoring interface in
> version 1.9. kdbus' version of eavesdropping is quite similar to the new
> monitoring interface.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Steven Rostedt]

On Wed, Apr 29, 2015 at 12:26:59PM -0400, John Stoffel wrote:
> 
> If your customers wnat this feature, you're more than welcome to fork
> the kernel and support it yourself.  Oh wait... Redhat does that
> already.  So what's the problem?   Just put it into RHEL (which I use
> I admit, along with Debian/Mint) and be done with it.

Red Hat tries very hard to push things upstream. It's policy is to not keep
things for themselves, but always work with the community. That way, everyone
benefits. Ideally, we should come up with a solution that works for all.

-- Steve

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Michal Hocko]

On Mon 27-04-15 13:11:03, Andy Lutomirski wrote:
> [resent without HTML]
> 
> On Apr 27, 2015 5:46 AM, "Michal Hocko"  wrote:
> >
> > On Wed 22-04-15 12:36:12, Andy Lutomirski wrote:
[...]
> > > The receiver gets to mmap the buffer.  I'm not sure what protection they 
> > > get.
> >
> > OK, so I've checked the code. kdbus_pool_new sets up a shmem file
> > (unlinked) so not visible externally. The consumer will get it via mmap
> > on the endpoint file by kdbus_pool_mmap and it refuses VM_WRITE and
> > clears VM_MAYWRITE. The receiver even doesn't have access to the shmem
> > file directly.
> >
> > It is ugly that kdbus_pool_mmap replaces the original vm_file and make
> > it point to the shmem file. I am not sure whether this is safe all the
> > time and it would deserve a big fat comment. On the other hand, it seems
> > some drivers are doing this already (e.g. dma_buf_mmap).
> 
> What happens to map_files in proc?  It seems unlikely that CRIU would
> ever work on dma_buf, but this could be a problem for CRIU with kdbus.

I am not familiar with CRIU and likewise with map_files directory. I've
actually heard about it for the first time.
[looking...]

So proc_map_files_readdir will iterate all VMAs including the one backed
by the buffer and it will see it's vm_file which will point to the shmem
file AFAICS. So it doesn't seem like CRIU would care because all parties
on the buffer would see the same inode. Whether that is really enough, I
dunno.
 
> > > The thing I'm worried about is that the receiver might deliberately
> > > avoid faulting in a bunch of pages and instead wait for the producer
> > > to touch them, causing pages that logically belong to the receiver to
> > > be charged to the producer instead.
> >
> > Hmm, now that I am looking into the code it seems you are right. E.g.
> > kdbus_cmd_send runs in the context of the sender AFAIU. This gets down
> > to kdbus_pool_slice_copy_iovec which does vfs_iter_write and this
> > is where we get to charge the memory. AFAIU the terminology all the
> > receivers will share the same shmem file when mmaping the endpoint.
> >
> > This, however, doesn't seem to be exploitable to hide memory charges
> > because the receiver cannot make the buffer writable. A nasty process
> > with a small memcg limit could still pre-fault the memory before any
> > writer gets sends a message and slow the whole endpoint traffic. But
> > that wouldn't be a completely new thing because processes might hammer
> > on memory even without memcg... It is just that this would be kind of
> > easier with memcg.
> > If that is the concern then the buffer should be pre-charged at the time
> > when it is created.
> 
> The attach I had in mind was that the nasty process with a small memcg
> creates one or many of these and doesn't pre-fault it. Then a sender
> (systemd?) sends messages and they get charged, possibly once for each
> copy sent, to the root memcg.

Dunno but I suspect that systemd will not talk to random endpoints. Or
can those endpoints be registered on a systembus by an untrusted task?
Bus vs. endpoint relation is still not entirely clear to me.

But even if that was possible I fail to see how the small memcg plays
any role when the task doesn't control the shmem buffer directly but it
only has a read only mapping of it.

> So kdbus should probably pre-charge the creator of the pool.

Yes, as I've said above.
-- 
Michal Hocko
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[David Lang]

On Wed, 29 Apr 2015, Martin Steigerwald wrote:


Am Mittwoch, 29. April 2015, 14:47:53 schrieb Harald Hoyer:

We really don't want the IPC mechanism to be in a flux state. All tools
have to fallback to a non-standard mechanism in that case.

If I have to pull in a dbus daemon in the initramfs, we still have the
chicken and egg problem for PID 1 talking to the logging daemon and
starting dbus.
systemd cannot talk to journald via dbus unless dbus-daemon is started,
dbus cannot log anything on startup, if journald is not running, etc...


Do I get this right that it is basically a userspace *design* decision
that you use as a reason to have kdbus inside the kernel?

Is it really necessary to use DBUS for talking to journald? And does it
really matter that much if any message before starting up dbus do not
appear in the log? /proc/kmsg is a ring buffer, it can still be copied over
later.


I've been getting the early boot messages in my logs for decades (assuming the 
system doesn't fail before the syslog daemon is started). It sometimes has 
required setting a larger than default ringbuffer in the kernel, but that's easy 
enough to do.


David Lang
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[John Stoffel]

> "Harald" == Harald Hoyer  writes:

Harald> On 29.04.2015 15:33, Richard Weinberger wrote:
>> It depends how you define "beginning". To me an initramfs is a *very* minimal
>> tool to prepare the rootfs and nothing more (no udev, no systemd, no
>> "mini distro").
>> If the initramfs fails to do its job it can print to the console like
>> the kernel does if it fails
>> at a very early stage.
>> 

Harald> Your solution might work for your small personal needs, but
Harald> not for our customers.

Arguing that your needs outweight mine because you have customers
ain't gonna fly... I don't care about your customers and why should I?
I'm not getting any money from them.  Nor do I make any money from
Linux kernel though as an IT person, I support Linux all day long.  Do
my requirements get listened to as well?

If your customers wnat this feature, you're more than welcome to fork
the kernel and support it yourself.  Oh wait... Redhat does that
already.  So what's the problem?   Just put it into RHEL (which I use
I admit, along with Debian/Mint) and be done with it.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Martin Steigerwald]

Am Mittwoch, 29. April 2015, 11:03:41 schrieb Theodore Ts'o:
> On Wed, Apr 29, 2015 at 04:53:53PM +0200, Harald Hoyer wrote:
> > Sure, I can write one binary to rule them all, pull out all the code
> > from all tools I need, but for me an IPC mechanism sounds a lot
> > better. And it should be _one_ common IPC mechanism and not a
> > plethora of them. It should feel like an operating system and not
> > like a bunch of thrown together software, which is glued together
> > with some magic shell scripts.
> 
> And so requiring wireshark (and X?) in initramfs to debug problems
> once dbus is introduced is better?
> 
> I would think shell scripts are *easier* to debug when things go
> wrong, especially in a minimal environment such as an initial ram
> disk.  Having had to debug problems in a distro initramfs when trying
> to help a customer bring up a FC boot disk long ago in another life,
> I'm certain I would rather debug problems while on site at a
> classified machine room[1] using shell scripts, and trying to debug
> dbus is something that would be infinitely worse.

Later in boot process I have seen some debugging issues with a systemd 
governed userspace:

Bug 1213778 - drops into emergency mode without any error message if it 
cannot find a filesystem in /etc/fstab
https://bugzilla.redhat.com/show_bug.cgi?id=1213778

Bug 1213781 - does not start ssh service if a filesystem in /etc/fstab 
cannot be mounted
https://bugzilla.redhat.com/show_bug.cgi?id=1213781

With shell scripts I had error messages for these, here I had to browse 
the output of journalctl -xb to find out.

And then what do I do if dbus communication is not working for some 
reason? Will I then be able to use journalctl at all?

Not that proper error reporting can´t be added in systemd and the 
dependency handling can´t be fixed towards some more sanity like for 
example "no /boot mount, no worries, I still start that ssh service for 
you", but currently for me this is a clear and heavy regression when I 
compare this with sysvinit boot behavior.

I do fix things when being dropped to initramfs shell, I added some script 
snippet for supporting BTRFS RAID 1 while it wasn´t yet supported in 
Debian (dunno if it is for booting in Jessie, but as my bug reports have 
not been closed yet, maybe bot), I still want to be able to do these kinds 
of things.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Austin S Hemmelgarn]

On 2015-04-29 11:22, Harald Hoyer wrote:

On 29.04.2015 17:17, Austin S Hemmelgarn wrote:

On 2015-04-29 11:07, Harald Hoyer wrote:

Most of the stuff does not work without udev and something like systemd.


That's funny, apparently the initramfs images I've been using for multiple
months now on server systems at work which don't have systemd, udev, or dbus,
and do LVM/RAID assembly, network configuration, crypto devices, multipath,
many different filesystems, and a number of other oddball configurations due to
the insanity that is the software I have to deal with from our company, don't
work.  I wonder how my systems are booting successfully 100% of the time then?




Then you should probably open source your initramfs, so we can all benefit from
it and use it for all distributions.

It's (mostly, aside from a couple of overlays to deal with the hoops I 
have to jump through to get some of our software working) just the 
standard one generated by Gentoo's 'genkernel' program (specifically the 
version from the genkernel-ng package), although now that I actually 
look at it, it might have udev in it, although I'm certain the ones that 
I have don't have systemd or dbus.




smime.p7s
Description: S/MIME Cryptographic Signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Martin Steigerwald]

Am Mittwoch, 29. April 2015, 14:47:53 schrieb Harald Hoyer:
> We really don't want the IPC mechanism to be in a flux state. All tools
> have to fallback to a non-standard mechanism in that case.
> 
> If I have to pull in a dbus daemon in the initramfs, we still have the
> chicken and egg problem for PID 1 talking to the logging daemon and
> starting dbus.
> systemd cannot talk to journald via dbus unless dbus-daemon is started,
> dbus cannot log anything on startup, if journald is not running, etc...

Do I get this right that it is basically a userspace *design* decision 
that you use as a reason to have kdbus inside the kernel?

Is it really necessary to use DBUS for talking to journald? And does it 
really matter that much if any message before starting up dbus do not 
appear in the log? /proc/kmsg is a ring buffer, it can still be copied over 
later.

I remember this kind of reason not not having cgroup management in a 
separate process, but these are both in userspace.

"We have done it this way in userspace, thus this needs to be in kernel" 
doesn´t sound quite convincing to me as an argument for having dbus inside 
the kernel. Userspace uses the API the kernel and glibc provide, yes, it 
makes sense to look at what userspace needs, but designing some things in 
userspace and then requiring support for these design decisions in the 
kernel just doesn´t sound quite right to me.

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Harald Hoyer]

On 29.04.2015 17:17, Austin S Hemmelgarn wrote:
> On 2015-04-29 11:07, Harald Hoyer wrote:
>> Most of the stuff does not work without udev and something like systemd.
>>
> That's funny, apparently the initramfs images I've been using for multiple
> months now on server systems at work which don't have systemd, udev, or dbus,
> and do LVM/RAID assembly, network configuration, crypto devices, multipath,
> many different filesystems, and a number of other oddball configurations due 
> to
> the insanity that is the software I have to deal with from our company, don't
> work.  I wonder how my systems are booting successfully 100% of the time then?
> 
> 

Then you should probably open source your initramfs, so we can all benefit from
it and use it for all distributions.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Austin S Hemmelgarn]

On 2015-04-29 11:03, Theodore Ts'o wrote:

On Wed, Apr 29, 2015 at 04:53:53PM +0200, Harald Hoyer wrote:

Sure, I can write one binary to rule them all, pull out all the code from all
tools I need, but for me an IPC mechanism sounds a lot better. And it should be
_one_ common IPC mechanism and not a plethora of them. It should feel like an
operating system and not like a bunch of thrown together software, which is
glued together with some magic shell scripts.


And so requiring wireshark (and X?) in initramfs to debug problems
once dbus is introduced is better?

I would think shell scripts are *easier* to debug when things go
wrong, especially in a minimal environment such as an initial ram
disk.  Having had to debug problems in a distro initramfs when trying
to help a customer bring up a FC boot disk long ago in another life,
I'm certain I would rather debug problems while on site at a
classified machine room[1] using shell scripts, and trying to debug
dbus is something that would be infinitely worse.

- Ted

[1] So no laptop, no google, no access to sources to figure out random
dbus messages, etc.

Likewise.

I keep hearing from people that shell scripting is hard, it really isn't 
compared to a number of other scripting languages, you just need to 
actually learn to do it right (which is getting more and more difficult 
these days cause fewer and fewer CS schools are teaching Unix).




smime.p7s
Description: S/MIME Cryptographic Signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Austin S Hemmelgarn]

On 2015-04-29 11:07, Harald Hoyer wrote:

Most of the stuff does not work without udev and something like systemd.

That's funny, apparently the initramfs images I've been using for 
multiple months now on server systems at work which don't have systemd, 
udev, or dbus, and do LVM/RAID assembly, network configuration, crypto 
devices, multipath, many different filesystems, and a number of other 
oddball configurations due to the insanity that is the software I have 
to deal with from our company, don't work.  I wonder how my systems are 
booting successfully 100% of the time then?





smime.p7s
Description: S/MIME Cryptographic Signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Simon McVittie]

On 29/04/15 14:35, Stephen Smalley wrote:
> As it currently stands, there
> are no LSM hook calls in the kdbus tree beyond metadata collection of
> security labels.

SELinux and AppArmor are the two particularly interesting LSMs here:
those are the ones that have support for user-space mediation in
dbus-daemon, and hence the ones for which replacing dbus-daemon with
kdbus, without LSM hooks, would be a regression.

> It is also interesting that kdbus allows impersonation of any
> credential, including security label, by "privileged" clients, where
> privileged simply means it either has CAP_IPC_OWNER or owns (euid
> matches uid) the bus.

FWIW, this particular feature is *not* one of those that are necessary
for feature parity with dbus-daemon. There's no API for making
dbus-daemon fake its clients' credentials; if you can ptrace it, then
you can of course subvert it arbitrarily, but nothing less hackish than
that is currently offered.

> On 04/29/2015 08:47 AM, Harald Hoyer wrote:
>>  * Eavesdropping on the kernel level, so privileged users can hook into
>>the message stream without hacking support for that into their
>>userspace processes
> 
> This one worried me a bit, particularly the statement that such
> eavesdropping is unobservable by any other participant on the bus.
> Seems a bit prone to abuse, particularly since it can be done by any
> privileged client, not merely the process that originally created the bus?

For feature parity with dbus-daemon, the fact that
eavesdropping/monitoring *exists* is necessary (it's a widely used
developer/sysadmin feature) but the precise mechanics of how you get it
are not necessarily set in stone. In particular, if you think kdbus'
definition of "are you privileged?" may be too broad, that seems a valid
question to be asking.

In traditional D-Bus, individual users can normally eavesdrop/monitor on
their own session buses (which are not a security boundary, unless
specially reconfigured), and this is a useful property; on non-LSM
systems without special configuration, each user should ideally be able
to monitor their own kdbus user bus, too.

The system bus *is* a security boundary, and administrative privileges
should be required to eavesdrop on it. At a high level, someone with
"full root privileges" should be able to eavesdrop, and ordinary users
should not; there are various possible criteria for distinguishing
between those two extremes, and I have no opinion on whether
CAP_IPC_OWNER is the most appropriate cutoff point.

In dbus-daemon, LSMs with integration code in dbus-daemon have the
opportunity to mediate eavesdropping specially. SELinux does not
currently do this (as far as I can see), but AppArmor does, so
AppArmor-confined processes are not normally allowed to eavesdrop on the
session bus (even though the same user's unconfined processes may). That
seems like one of the obvious places for an LSM hook in kdbus.

Having eavesdropping be unobservable means that applications cannot
change their behaviour while they are being watched, either maliciously
(to hide from investigation) or accidentally (bugs that only happen when
not being debugged are the hardest to fix). dbus-daemon's traditional
implementation of eavesdropping has had side-effects in the past, which
is undesirable, and is addressed by the new monitoring interface in
version 1.9. kdbus' version of eavesdropping is quite similar to the new
monitoring interface.

-- 
Simon McVittie
Collabora Ltd. 
For context, I am a D-Bus maintainer, but neither the original designer
of D-Bus nor a kdbus developer.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Harald Hoyer]

On 29.04.2015 16:46, Austin S Hemmelgarn wrote:
> On 2015-04-29 10:11, Harald Hoyer wrote:
>> On 29.04.2015 16:04, Richard Weinberger wrote:
>>> Am 29.04.2015 um 16:01 schrieb Harald Hoyer:
 On 29.04.2015 15:46, Richard Weinberger wrote:
> Am 29.04.2015 um 15:38 schrieb Harald Hoyer:
>> On 29.04.2015 15:33, Richard Weinberger wrote:
>>> It depends how you define "beginning". To me an initramfs is a *very*
>>> minimal
>>> tool to prepare the rootfs and nothing more (no udev, no systemd, no
>>> "mini distro").
>>> If the initramfs fails to do its job it can print to the console like
>>> the kernel does if it fails
>>> at a very early stage.
>>>
>>
>> Your solution might work for your small personal needs, but not for our
>> customers.
>
> Correct, I don't know your customers, all I know are my customers. :-)
>
> What feature do your customers need?
> I mean, I fully agree with you that an initramfs must not fail silently
> but how does dbus help there? If it fails to mount the rootfs there is not
> much it can do.
>
> Thanks,
> //richard
>

 We don't handcraft the initramfs script for every our customers, therefore 
 we
 have to generically support hotplug, persistent device names, persistent
 interface names, network connectivity in the initramfs, user input handling
 for
 passwords, fonts, keyboard layouts, fips, fsck, repair tools for file 
 systems,
 raid assembly, LVM assembly, multipath, crypto devices, live images, iSCSI,
 FCoE, all kinds of filesystems with their quirks, IBM z-series support, 
 resume
 from hibernation, […]
>>>
>>> This is correct. But which of these tools/features depend on dbus?
>>
>> I would love to add dbus support to all of them and use it, so I can connect
>> them all more easily. No need for them to invent their own version of IPC,
>> which can only be used by their own tool set.
>>
> Resume is built into the kernel, so no need for IPC there.  Keymaps, fonts, 
> and
> fsck need no IPC either.  FIPS related stuff should need no IPC.  Anything to
> do with the Device Mapper and hotplug should just need uevents.  While I can
> kind of see you wanting to have lvmetad in the initramfs for use with LVM, 
> I've
> seen all kinds of reports of issues caused by that.  I can also kind of
> understand wanting some kind of unified IPC for the netboot related stuff, 
> DBus
> is still serious overkill for any of that IMHO.  As things stand currently, 
> the
> few things in that list that I know actually use IPC for anything get by just
> fine (and much faster) using just UDS.
> 
> 

He asked what customers need, because he does not need udev, systemd, "mini
distro".

Most of the stuff does not work without udev and something like systemd.

And all of the stuff I mentioned together forms a "mini distro" for me.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Theodore Ts'o]

On Wed, Apr 29, 2015 at 04:53:53PM +0200, Harald Hoyer wrote:
> Sure, I can write one binary to rule them all, pull out all the code from all
> tools I need, but for me an IPC mechanism sounds a lot better. And it should 
> be
> _one_ common IPC mechanism and not a plethora of them. It should feel like an
> operating system and not like a bunch of thrown together software, which is
> glued together with some magic shell scripts.

And so requiring wireshark (and X?) in initramfs to debug problems
once dbus is introduced is better?

I would think shell scripts are *easier* to debug when things go
wrong, especially in a minimal environment such as an initial ram
disk.  Having had to debug problems in a distro initramfs when trying
to help a customer bring up a FC boot disk long ago in another life,
I'm certain I would rather debug problems while on site at a
classified machine room[1] using shell scripts, and trying to debug
dbus is something that would be infinitely worse.

- Ted

[1] So no laptop, no google, no access to sources to figure out random
dbus messages, etc.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Richard Weinberger]

Am 29.04.2015 um 16:53 schrieb Harald Hoyer:
> On 29.04.2015 16:18, Richard Weinberger wrote:
>> Am 29.04.2015 um 16:11 schrieb Harald Hoyer:
> We don't handcraft the initramfs script for every our customers, 
> therefore we
> have to generically support hotplug, persistent device names, persistent
> interface names, network connectivity in the initramfs, user input 
> handling for
> passwords, fonts, keyboard layouts, fips, fsck, repair tools for file 
> systems,
> raid assembly, LVM assembly, multipath, crypto devices, live images, 
> iSCSI,
> FCoE, all kinds of filesystems with their quirks, IBM z-series support, 
> resume
> from hibernation, […]

 This is correct. But which of these tools/features depend on dbus?
>>>
>>> I would love to add dbus support to all of them and use it, so I can connect
>>> them all more easily. No need for them to invent their own version of IPC,
>>> which can only be used by their own tool set.
>>
>> Why/how do you need to connect them?
>> Sorry for being persistent but as I use most of these tools too (also in 
>> initramfs)
>> I'm very curious.
>>
>> Many of us grumpy kernel devs simply don't know all the use case of you have 
>> to cover.
>> So, please explain. :-)
>>
> 
> Well, using shell scripts I connected all of these tools in the earlier
> versions of dracut [1]. Been there, done that.
> 
> When using bash to wait for an interface to come up [2] or doing dhcp [3], the
> (at least my) pain threshold is reached, and you want something more 
> sophisticated.
> 
> So, one starts eyeing NetworkManager or systemd-networkd. Both of them have 
> CLI
> tools and helpers and these tools and helpers talk to each other with (guess
> what?) an IPC mechanism, which happens to be DBUS (because it's the IPC of
> choice, if you don't want to reinvent the wheel).
> 
> But let's not pinpoint that to network alone. Parsing output of tools with
> shell scripts is horrible, slow, fragile, error prone.

So, you want to replace bash by dbus?
I'll stop now with arguing.
Let's agree to disagree.

> Sure, I can write one binary to rule them all, pull out all the code from all
> tools I need, but for me an IPC mechanism sounds a lot better. And it should 
> be
> _one_ common IPC mechanism and not a plethora of them. It should feel like an
> operating system and not like a bunch of thrown together software, which is
> glued together with some magic shell scripts.

This is how UNIX works. ;)

Thanks,
//richard
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Harald Hoyer]

On 29.04.2015 16:18, Richard Weinberger wrote:
> Am 29.04.2015 um 16:11 schrieb Harald Hoyer:
 We don't handcraft the initramfs script for every our customers, therefore 
 we
 have to generically support hotplug, persistent device names, persistent
 interface names, network connectivity in the initramfs, user input 
 handling for
 passwords, fonts, keyboard layouts, fips, fsck, repair tools for file 
 systems,
 raid assembly, LVM assembly, multipath, crypto devices, live images, iSCSI,
 FCoE, all kinds of filesystems with their quirks, IBM z-series support, 
 resume
 from hibernation, […]
>>>
>>> This is correct. But which of these tools/features depend on dbus?
>>
>> I would love to add dbus support to all of them and use it, so I can connect
>> them all more easily. No need for them to invent their own version of IPC,
>> which can only be used by their own tool set.
> 
> Why/how do you need to connect them?
> Sorry for being persistent but as I use most of these tools too (also in 
> initramfs)
> I'm very curious.
> 
> Many of us grumpy kernel devs simply don't know all the use case of you have 
> to cover.
> So, please explain. :-)
> 

Well, using shell scripts I connected all of these tools in the earlier
versions of dracut [1]. Been there, done that.

When using bash to wait for an interface to come up [2] or doing dhcp [3], the
(at least my) pain threshold is reached, and you want something more 
sophisticated.

So, one starts eyeing NetworkManager or systemd-networkd. Both of them have CLI
tools and helpers and these tools and helpers talk to each other with (guess
what?) an IPC mechanism, which happens to be DBUS (because it's the IPC of
choice, if you don't want to reinvent the wheel).

But let's not pinpoint that to network alone. Parsing output of tools with
shell scripts is horrible, slow, fragile, error prone.

Sure, I can write one binary to rule them all, pull out all the code from all
tools I need, but for me an IPC mechanism sounds a lot better. And it should be
_one_ common IPC mechanism and not a plethora of them. It should feel like an
operating system and not like a bunch of thrown together software, which is
glued together with some magic shell scripts.


[1] https://dracut.wiki.kernel.org/index.php/Main_Page
[2]
http://git.kernel.org/cgit/boot/dracut/dracut.git/tree/modules.d/40network/net-lib.sh#n483
[3] http://pkgs.fedoraproject.org/cgit/dhcp.git/tree/dhclient-script
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Richard Weinberger]

Am 29.04.2015 um 16:46 schrieb Austin S Hemmelgarn:
> On 2015-04-29 10:11, Harald Hoyer wrote:
>> On 29.04.2015 16:04, Richard Weinberger wrote:
>>> Am 29.04.2015 um 16:01 schrieb Harald Hoyer:
 On 29.04.2015 15:46, Richard Weinberger wrote:
> Am 29.04.2015 um 15:38 schrieb Harald Hoyer:
>> On 29.04.2015 15:33, Richard Weinberger wrote:
>>> It depends how you define "beginning". To me an initramfs is a *very* 
>>> minimal
>>> tool to prepare the rootfs and nothing more (no udev, no systemd, no
>>> "mini distro").
>>> If the initramfs fails to do its job it can print to the console like
>>> the kernel does if it fails
>>> at a very early stage.
>>>
>>
>> Your solution might work for your small personal needs, but not for our 
>> customers.
>
> Correct, I don't know your customers, all I know are my customers. :-)
>
> What feature do your customers need?
> I mean, I fully agree with you that an initramfs must not fail silently
> but how does dbus help there? If it fails to mount the rootfs there is not
> much it can do.
>
> Thanks,
> //richard
>

 We don't handcraft the initramfs script for every our customers, therefore 
 we
 have to generically support hotplug, persistent device names, persistent
 interface names, network connectivity in the initramfs, user input 
 handling for
 passwords, fonts, keyboard layouts, fips, fsck, repair tools for file 
 systems,
 raid assembly, LVM assembly, multipath, crypto devices, live images, iSCSI,
 FCoE, all kinds of filesystems with their quirks, IBM z-series support, 
 resume
 from hibernation, […]
>>>
>>> This is correct. But which of these tools/features depend on dbus?
>>
>> I would love to add dbus support to all of them and use it, so I can connect
>> them all more easily. No need for them to invent their own version of IPC,
>> which can only be used by their own tool set.
>>
> Resume is built into the kernel, so no need for IPC there.  Keymaps, fonts, 
> and fsck need no IPC either.  FIPS related stuff should need no IPC.  
> Anything to do with the Device
> Mapper and hotplug should just need uevents.  While I can kind of see you 
> wanting to have lvmetad in the initramfs for use with LVM, I've seen all 
> kinds of reports of issues caused
> by that.  I can also kind of understand wanting some kind of unified IPC for 
> the netboot related stuff, DBus is still serious overkill for any of that 
> IMHO.  As things stand
> currently, the few things in that list that I know actually use IPC for 
> anything get by just fine (and much faster) using just UDS.

Even if dbus is really needed you can embed it into the initramfs.
What you need is a good bootstrap procedure between dbus/systemd within the 
initramfs and the real rootfs.

Thanks,
//richard
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Austin S Hemmelgarn]

On 2015-04-29 10:11, Harald Hoyer wrote:

On 29.04.2015 16:04, Richard Weinberger wrote:

Am 29.04.2015 um 16:01 schrieb Harald Hoyer:

On 29.04.2015 15:46, Richard Weinberger wrote:

Am 29.04.2015 um 15:38 schrieb Harald Hoyer:

On 29.04.2015 15:33, Richard Weinberger wrote:

It depends how you define "beginning". To me an initramfs is a *very* minimal
tool to prepare the rootfs and nothing more (no udev, no systemd, no
"mini distro").
If the initramfs fails to do its job it can print to the console like
the kernel does if it fails
at a very early stage.



Your solution might work for your small personal needs, but not for our 
customers.


Correct, I don't know your customers, all I know are my customers. :-)

What feature do your customers need?
I mean, I fully agree with you that an initramfs must not fail silently
but how does dbus help there? If it fails to mount the rootfs there is not
much it can do.

Thanks,
//richard



We don't handcraft the initramfs script for every our customers, therefore we
have to generically support hotplug, persistent device names, persistent
interface names, network connectivity in the initramfs, user input handling for
passwords, fonts, keyboard layouts, fips, fsck, repair tools for file systems,
raid assembly, LVM assembly, multipath, crypto devices, live images, iSCSI,
FCoE, all kinds of filesystems with their quirks, IBM z-series support, resume
from hibernation, […]


This is correct. But which of these tools/features depend on dbus?


I would love to add dbus support to all of them and use it, so I can connect
them all more easily. No need for them to invent their own version of IPC,
which can only be used by their own tool set.

Resume is built into the kernel, so no need for IPC there.  Keymaps, 
fonts, and fsck need no IPC either.  FIPS related stuff should need no 
IPC.  Anything to do with the Device Mapper and hotplug should just need 
uevents.  While I can kind of see you wanting to have lvmetad in the 
initramfs for use with LVM, I've seen all kinds of reports of issues 
caused by that.  I can also kind of understand wanting some kind of 
unified IPC for the netboot related stuff, DBus is still serious 
overkill for any of that IMHO.  As things stand currently, the few 
things in that list that I know actually use IPC for anything get by 
just fine (and much faster) using just UDS.





smime.p7s
Description: S/MIME Cryptographic Signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Richard Weinberger]

Am 29.04.2015 um 16:11 schrieb Harald Hoyer:
>>> We don't handcraft the initramfs script for every our customers, therefore 
>>> we
>>> have to generically support hotplug, persistent device names, persistent
>>> interface names, network connectivity in the initramfs, user input handling 
>>> for
>>> passwords, fonts, keyboard layouts, fips, fsck, repair tools for file 
>>> systems,
>>> raid assembly, LVM assembly, multipath, crypto devices, live images, iSCSI,
>>> FCoE, all kinds of filesystems with their quirks, IBM z-series support, 
>>> resume
>>> from hibernation, […]
>>
>> This is correct. But which of these tools/features depend on dbus?
> 
> I would love to add dbus support to all of them and use it, so I can connect
> them all more easily. No need for them to invent their own version of IPC,
> which can only be used by their own tool set.

Why/how do you need to connect them?
Sorry for being persistent but as I use most of these tools too (also in 
initramfs)
I'm very curious.

Many of us grumpy kernel devs simply don't know all the use case of you have to 
cover.
So, please explain. :-)

Thanks,
//richard
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Harald Hoyer]

On 29.04.2015 16:04, Richard Weinberger wrote:
> Am 29.04.2015 um 16:01 schrieb Harald Hoyer:
>> On 29.04.2015 15:46, Richard Weinberger wrote:
>>> Am 29.04.2015 um 15:38 schrieb Harald Hoyer:
 On 29.04.2015 15:33, Richard Weinberger wrote:
> It depends how you define "beginning". To me an initramfs is a *very* 
> minimal
> tool to prepare the rootfs and nothing more (no udev, no systemd, no
> "mini distro").
> If the initramfs fails to do its job it can print to the console like
> the kernel does if it fails
> at a very early stage.
>

 Your solution might work for your small personal needs, but not for our 
 customers.
>>>
>>> Correct, I don't know your customers, all I know are my customers. :-)
>>>
>>> What feature do your customers need?
>>> I mean, I fully agree with you that an initramfs must not fail silently
>>> but how does dbus help there? If it fails to mount the rootfs there is not
>>> much it can do.
>>>
>>> Thanks,
>>> //richard
>>>
>>
>> We don't handcraft the initramfs script for every our customers, therefore we
>> have to generically support hotplug, persistent device names, persistent
>> interface names, network connectivity in the initramfs, user input handling 
>> for
>> passwords, fonts, keyboard layouts, fips, fsck, repair tools for file 
>> systems,
>> raid assembly, LVM assembly, multipath, crypto devices, live images, iSCSI,
>> FCoE, all kinds of filesystems with their quirks, IBM z-series support, 
>> resume
>> from hibernation, […]
> 
> This is correct. But which of these tools/features depend on dbus?

I would love to add dbus support to all of them and use it, so I can connect
them all more easily. No need for them to invent their own version of IPC,
which can only be used by their own tool set.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Richard Weinberger]

Am 29.04.2015 um 16:01 schrieb Harald Hoyer:
> On 29.04.2015 15:46, Richard Weinberger wrote:
>> Am 29.04.2015 um 15:38 schrieb Harald Hoyer:
>>> On 29.04.2015 15:33, Richard Weinberger wrote:
 It depends how you define "beginning". To me an initramfs is a *very* 
 minimal
 tool to prepare the rootfs and nothing more (no udev, no systemd, no
 "mini distro").
 If the initramfs fails to do its job it can print to the console like
 the kernel does if it fails
 at a very early stage.

>>>
>>> Your solution might work for your small personal needs, but not for our 
>>> customers.
>>
>> Correct, I don't know your customers, all I know are my customers. :-)
>>
>> What feature do your customers need?
>> I mean, I fully agree with you that an initramfs must not fail silently
>> but how does dbus help there? If it fails to mount the rootfs there is not
>> much it can do.
>>
>> Thanks,
>> //richard
>>
> 
> We don't handcraft the initramfs script for every our customers, therefore we
> have to generically support hotplug, persistent device names, persistent
> interface names, network connectivity in the initramfs, user input handling 
> for
> passwords, fonts, keyboard layouts, fips, fsck, repair tools for file systems,
> raid assembly, LVM assembly, multipath, crypto devices, live images, iSCSI,
> FCoE, all kinds of filesystems with their quirks, IBM z-series support, resume
> from hibernation, […]

This is correct. But which of these tools/features depend on dbus?

Thanks,
//richard

P.s: Please don't drop the CC list.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Harald Hoyer]

On 29.04.2015 15:46, Richard Weinberger wrote:
> Am 29.04.2015 um 15:38 schrieb Harald Hoyer:
>> On 29.04.2015 15:33, Richard Weinberger wrote:
>>> It depends how you define "beginning". To me an initramfs is a *very* 
>>> minimal
>>> tool to prepare the rootfs and nothing more (no udev, no systemd, no
>>> "mini distro").
>>> If the initramfs fails to do its job it can print to the console like
>>> the kernel does if it fails
>>> at a very early stage.
>>>
>>
>> Your solution might work for your small personal needs, but not for our 
>> customers.
> 
> Correct, I don't know your customers, all I know are my customers. :-)
> 
> What feature do your customers need?
> I mean, I fully agree with you that an initramfs must not fail silently
> but how does dbus help there? If it fails to mount the rootfs there is not
> much it can do.
> 
> Thanks,
> //richard
> 

We don't handcraft the initramfs script for every our customers, therefore we
have to generically support hotplug, persistent device names, persistent
interface names, network connectivity in the initramfs, user input handling for
passwords, fonts, keyboard layouts, fips, fsck, repair tools for file systems,
raid assembly, LVM assembly, multipath, crypto devices, live images, iSCSI,
FCoE, all kinds of filesystems with their quirks, IBM z-series support, resume
from hibernation, […]

And no, this is not a simple minimal tool.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Richard Weinberger]

Am 29.04.2015 um 15:38 schrieb Harald Hoyer:
> On 29.04.2015 15:33, Richard Weinberger wrote:
>> It depends how you define "beginning". To me an initramfs is a *very* minimal
>> tool to prepare the rootfs and nothing more (no udev, no systemd, no
>> "mini distro").
>> If the initramfs fails to do its job it can print to the console like
>> the kernel does if it fails
>> at a very early stage.
>>
> 
> Your solution might work for your small personal needs, but not for our 
> customers.

Correct, I don't know your customers, all I know are my customers. :-)

What feature do your customers need?
I mean, I fully agree with you that an initramfs must not fail silently
but how does dbus help there? If it fails to mount the rootfs there is not
much it can do.

Thanks,
//richard
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Harald Hoyer]

On 29.04.2015 15:33, Richard Weinberger wrote:
> It depends how you define "beginning". To me an initramfs is a *very* minimal
> tool to prepare the rootfs and nothing more (no udev, no systemd, no
> "mini distro").
> If the initramfs fails to do its job it can print to the console like
> the kernel does if it fails
> at a very early stage.
> 

Your solution might work for your small personal needs, but not for our 
customers.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Stephen Smalley]

On 04/29/2015 08:47 AM, Harald Hoyer wrote:
> On 29.04.2015 01:12, John Stoffel wrote:
>> LDAP is pretty damn generic, in that you can put pretty large objects into
>> it, and pretty large OUs, etc.  So why would it be a candidate for going
>> into the kernel?  And why is kdbus so important in the kernel as well?
>> People have talked about it needing to be there for bootup, but isn't that
>> why we ripped out RAID detection and such from the kernel and built
>> initramfs, so that there's LESS in the kernel, and more in an early
>> userspace?  Same idea with dbus in my opinion.
> 
> Let me elaborate on the initramfs/shutdown situation a little bit more,
> because I have to deal with that every day.
> 
> Because of the "let's move everything to userspace" sentiment we nowadays
> have the situation, that we need a lot of tools to setup the root device.
> 
> Be it LVM on IMSM or iSCSI multipath, the initramfs has to setup the network
> (with bridging, bonding, etc.), the iSCSI connection, assemble the raid, the
> LVM, open crypto devices, etc...
> And if something goes wrong, you want to have a shell, see all the logs and
> debug things.
> 
> Now over the time we moved away from simple shell scripts (without any
> logging) and static compiled special versions for the initramfs to a mini
> distribution in the initramfs, which simplifies maintenance and improves
> reliability.
> 
> Basically you want to use the same tools in the initramfs (and shutdown)
> which you already have and use in your real root, with the same configuration
> files and the same interfaces and the same code paths.
> 
> Therefore systemd is started in dracut created initramfs, which starts
> journald for logging. The same basic systemd targets exist in the initramfs
> as on the real root, so normally you don't have to cope with specialized
> versions for the initramfs.
> 
> The target here is to have the same IPC mechanism from the very beginning to
> the very end. No crappy fallback mechanisms in case a daemon is not running
> or has crashed, no creepy transition from initramfs root to real root to
> shutdown root.
> 
> We already have such transitions like: systemd, journald, mdmon [1], etc.
> systemd has to serialize itself, journald's file descriptors are transitioned
> over, mdmon jumps through hoops. Remember you want to get rid of open files
> and executables and have to reexec everything, if you transition from the
> initramfs root to the real root, and also from the real root to the shutdown
> root.
> 
> We really don't want the IPC mechanism to be in a flux state. All tools have
> to fallback to a non-standard mechanism in that case.
> 
> If I have to pull in a dbus daemon in the initramfs, we still have the
> chicken and egg problem for PID 1 talking to the logging daemon and starting
> dbus.
> systemd cannot talk to journald via dbus unless dbus-daemon is started, dbus
> cannot log anything on startup, if journald is not running, etc...
> 
> dbus-daemon would have to transition to the real root, and from the real root
> to the shutdown root, without losing state.
> 
> Of course this can all be done, but it would involve fallback mechanisms,
> which we want to get rid off. Hopefully, you don't suggest to merge dbus with
> PID 1. Also with a daemon, you will lose the points mentioned in the cover 
> mail
> :
> 
>  * Security: The peers which communicate do not have to trust each
>other, as the only trustworthy component in the game is the kernel
>which adds metadata and ensures that all data passed as payload is
>either copied or sealed, so that the receiver can parse the data
>without having to protect against changing memory while parsing
>buffers. Also, all the data transfer is controlled by the kernel,
>so that LSMs can track and control what is going on, without
>involving userspace. Because of the LSM issue, security people are
>much happier with this model than the current scheme of having to
>hook into dbus to mediate things.

I just want to caution that this justification for kdbus is not fully
realized in the current implementation.  As it currently stands, there
are no LSM hook calls in the kdbus tree beyond metadata collection of
security labels.  I know there have been experimental proof-of-concept
patches floating around for LSM hooks for kdbus but they aren't merged
as of yet, nor are there any real implementations of the hooks for the
security modules.  If kdbus is merged, we need to make sure that the LSM
support is integrated before it gets enabled in any distros or we'll
have a completely unmediated IPC channel, i.e. a bypass of any security
policy restrictions on IPC.

It is also interesting that kdbus allows impersonation of any
credential, including security label, by "privileged" clients, where
privileged simply means it either has CAP_IPC_OWNER or owns (euid
matches uid) the bus.  That seems wrong at least for security labels;
either we should not support impersonation 

Re: [GIT PULL] kdbus for 4.1-rc1

[Richard Weinberger]

On Wed, Apr 29, 2015 at 2:47 PM, Harald Hoyer  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> On 29.04.2015 01:12, John Stoffel wrote:
>> LDAP is pretty damn generic, in that you can put pretty large objects into
>> it, and pretty large OUs, etc.  So why would it be a candidate for going
>> into the kernel?  And why is kdbus so important in the kernel as well?
>> People have talked about it needing to be there for bootup, but isn't that
>> why we ripped out RAID detection and such from the kernel and built
>> initramfs, so that there's LESS in the kernel, and more in an early
>> userspace?  Same idea with dbus in my opinion.
>
> Let me elaborate on the initramfs/shutdown situation a little bit more,
> because I have to deal with that every day.
>
> Because of the "let's move everything to userspace" sentiment we nowadays
> have the situation, that we need a lot of tools to setup the root device.
>
> Be it LVM on IMSM or iSCSI multipath, the initramfs has to setup the network
> (with bridging, bonding, etc.), the iSCSI connection, assemble the raid, the
> LVM, open crypto devices, etc...
> And if something goes wrong, you want to have a shell, see all the logs and
> debug things.

None of these tools depend on dbus (_desktop_ bus).

> Now over the time we moved away from simple shell scripts (without any
> logging) and static compiled special versions for the initramfs to a mini
> distribution in the initramfs, which simplifies maintenance and improves
> reliability.
>
> Basically you want to use the same tools in the initramfs (and shutdown)
> which you already have and use in your real root, with the same configuration
> files and the same interfaces and the same code paths.
>
> Therefore systemd is started in dracut created initramfs, which starts
> journald for logging. The same basic systemd targets exist in the initramfs
> as on the real root, so normally you don't have to cope with specialized
> versions for the initramfs.
>
> The target here is to have the same IPC mechanism from the very beginning to
> the very end. No crappy fallback mechanisms in case a daemon is not running
> or has crashed, no creepy transition from initramfs root to real root to
> shutdown root.
>
> We already have such transitions like: systemd, journald, mdmon [1], etc.
> systemd has to serialize itself, journald's file descriptors are transitioned
> over, mdmon jumps through hoops. Remember you want to get rid of open files
> and executables and have to reexec everything, if you transition from the
> initramfs root to the real root, and also from the real root to the shutdown
> root.
>
> We really don't want the IPC mechanism to be in a flux state. All tools have
> to fallback to a non-standard mechanism in that case.
>
> If I have to pull in a dbus daemon in the initramfs, we still have the
> chicken and egg problem for PID 1 talking to the logging daemon and starting
> dbus.
> systemd cannot talk to journald via dbus unless dbus-daemon is started, dbus
> cannot log anything on startup, if journald is not running, etc...

The only reason why you need dbus in your initramfs is because of systemd
and its tools, isn't it?

> In my ideal world, there is a standard IPC mechanism from the beginning to
> the end, which does not rely on any process running (except the kernel) and
> which is used by _all_ tools, be it a system daemon providing information and
> interfaces about device assembly or network setup tools or end user desktop
> processes.

It depends how you define "beginning". To me an initramfs is a *very* minimal
tool to prepare the rootfs and nothing more (no udev, no systemd, no
"mini distro").
If the initramfs fails to do its job it can print to the console like
the kernel does if it fails
at a very early stage.

-- 
Thanks,
//richard
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Harald Hoyer]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 29.04.2015 01:12, John Stoffel wrote:
> LDAP is pretty damn generic, in that you can put pretty large objects into
> it, and pretty large OUs, etc.  So why would it be a candidate for going
> into the kernel?  And why is kdbus so important in the kernel as well?
> People have talked about it needing to be there for bootup, but isn't that
> why we ripped out RAID detection and such from the kernel and built
> initramfs, so that there's LESS in the kernel, and more in an early
> userspace?  Same idea with dbus in my opinion.

Let me elaborate on the initramfs/shutdown situation a little bit more,
because I have to deal with that every day.

Because of the "let's move everything to userspace" sentiment we nowadays
have the situation, that we need a lot of tools to setup the root device.

Be it LVM on IMSM or iSCSI multipath, the initramfs has to setup the network
(with bridging, bonding, etc.), the iSCSI connection, assemble the raid, the
LVM, open crypto devices, etc...
And if something goes wrong, you want to have a shell, see all the logs and
debug things.

Now over the time we moved away from simple shell scripts (without any
logging) and static compiled special versions for the initramfs to a mini
distribution in the initramfs, which simplifies maintenance and improves
reliability.

Basically you want to use the same tools in the initramfs (and shutdown)
which you already have and use in your real root, with the same configuration
files and the same interfaces and the same code paths.

Therefore systemd is started in dracut created initramfs, which starts
journald for logging. The same basic systemd targets exist in the initramfs
as on the real root, so normally you don't have to cope with specialized
versions for the initramfs.

The target here is to have the same IPC mechanism from the very beginning to
the very end. No crappy fallback mechanisms in case a daemon is not running
or has crashed, no creepy transition from initramfs root to real root to
shutdown root.

We already have such transitions like: systemd, journald, mdmon [1], etc.
systemd has to serialize itself, journald's file descriptors are transitioned
over, mdmon jumps through hoops. Remember you want to get rid of open files
and executables and have to reexec everything, if you transition from the
initramfs root to the real root, and also from the real root to the shutdown
root.

We really don't want the IPC mechanism to be in a flux state. All tools have
to fallback to a non-standard mechanism in that case.

If I have to pull in a dbus daemon in the initramfs, we still have the
chicken and egg problem for PID 1 talking to the logging daemon and starting
dbus.
systemd cannot talk to journald via dbus unless dbus-daemon is started, dbus
cannot log anything on startup, if journald is not running, etc...

dbus-daemon would have to transition to the real root, and from the real root
to the shutdown root, without losing state.

Of course this can all be done, but it would involve fallback mechanisms,
which we want to get rid off. Hopefully, you don't suggest to merge dbus with
PID 1. Also with a daemon, you will lose the points mentioned in the cover mail
:

 * Security: The peers which communicate do not have to trust each
   other, as the only trustworthy component in the game is the kernel
   which adds metadata and ensures that all data passed as payload is
   either copied or sealed, so that the receiver can parse the data
   without having to protect against changing memory while parsing
   buffers. Also, all the data transfer is controlled by the kernel,
   so that LSMs can track and control what is going on, without
   involving userspace. Because of the LSM issue, security people are
   much happier with this model than the current scheme of having to
   hook into dbus to mediate things.

 * Being in the kernel closes a lot of races which can't be fixed with
   the current userspace solutions.  For example, with kdbus, there is a
   way a client can disconnect from a bus, but do so only if no further
   messages present in its queue, which is crucial for implementing
   race-free "exit-on-idle" services

 * Eavesdropping on the kernel level, so privileged users can hook into
   the message stream without hacking support for that into their
   userspace processes

 * A number of smaller benefits: for example kdbus learned a way to peek
   full messages without dequeing them, which is really useful for
   logging metadata when handling bus-activation requests.

I don't care, if the kdbus speedup is only marginal.

In my ideal world, there is a standard IPC mechanism from the beginning to
the end, which does not rely on any process running (except the kernel) and
which is used by _all_ tools, be it a system daemon providing information and
interfaces about device assembly or network setup tools or end user desktop
processes.

dbus _is_ such an easy, flexible standard IPC mechanism. Of 

Re: [GIT PULL] kdbus for 4.1-rc1

[Harald Hoyer]

On 29.04.2015 01:12, John Stoffel wrote:
>> "Havoc" == Havoc Pennington  writes:
> 
> Havoc> On Tue, Apr 28, 2015 at 1:18 PM, Theodore Ts'o  wrote:
>>> I find dbus to be extremely hard to debug when my desktop starts doing
>>> things I don't want it to do.  The fact that it might be flinging around 
>>> hundreds
>>> of thousands of messages, and that this is something we want to encourage,
> 
> Havoc> This particular argument doesn't resonate with me ... if dbus
> Havoc> is hard to debug, it's not as if "ad hoc application-specific
> Havoc> sidechannel somebody cooked up" is going to be easier.
> 
> When Ted is saying it's hard to debug... then maybe it's a bit crappy
> in design or implementation?  

There is a very nice tool to debug the traffic for kdbus.

http://lists.freedesktop.org/archives/dbus/2014-March/016178.html

Also the patched wireshark makes it as easy as analyzing network traffic.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Harald Hoyer]

On 29.04.2015 01:12, John Stoffel wrote:
 Havoc == Havoc Pennington h...@pobox.com writes:
 
 Havoc On Tue, Apr 28, 2015 at 1:18 PM, Theodore Ts'o ty...@mit.edu wrote:
 I find dbus to be extremely hard to debug when my desktop starts doing
 things I don't want it to do.  The fact that it might be flinging around 
 hundreds
 of thousands of messages, and that this is something we want to encourage,
 
 Havoc This particular argument doesn't resonate with me ... if dbus
 Havoc is hard to debug, it's not as if ad hoc application-specific
 Havoc sidechannel somebody cooked up is going to be easier.
 
 When Ted is saying it's hard to debug... then maybe it's a bit crappy
 in design or implementation?  

There is a very nice tool to debug the traffic for kdbus.

http://lists.freedesktop.org/archives/dbus/2014-March/016178.html

Also the patched wireshark makes it as easy as analyzing network traffic.
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Harald Hoyer]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 29.04.2015 01:12, John Stoffel wrote:
 LDAP is pretty damn generic, in that you can put pretty large objects into
 it, and pretty large OUs, etc.  So why would it be a candidate for going
 into the kernel?  And why is kdbus so important in the kernel as well?
 People have talked about it needing to be there for bootup, but isn't that
 why we ripped out RAID detection and such from the kernel and built
 initramfs, so that there's LESS in the kernel, and more in an early
 userspace?  Same idea with dbus in my opinion.

Let me elaborate on the initramfs/shutdown situation a little bit more,
because I have to deal with that every day.

Because of the let's move everything to userspace sentiment we nowadays
have the situation, that we need a lot of tools to setup the root device.

Be it LVM on IMSM or iSCSI multipath, the initramfs has to setup the network
(with bridging, bonding, etc.), the iSCSI connection, assemble the raid, the
LVM, open crypto devices, etc...
And if something goes wrong, you want to have a shell, see all the logs and
debug things.

Now over the time we moved away from simple shell scripts (without any
logging) and static compiled special versions for the initramfs to a mini
distribution in the initramfs, which simplifies maintenance and improves
reliability.

Basically you want to use the same tools in the initramfs (and shutdown)
which you already have and use in your real root, with the same configuration
files and the same interfaces and the same code paths.

Therefore systemd is started in dracut created initramfs, which starts
journald for logging. The same basic systemd targets exist in the initramfs
as on the real root, so normally you don't have to cope with specialized
versions for the initramfs.

The target here is to have the same IPC mechanism from the very beginning to
the very end. No crappy fallback mechanisms in case a daemon is not running
or has crashed, no creepy transition from initramfs root to real root to
shutdown root.

We already have such transitions like: systemd, journald, mdmon [1], etc.
systemd has to serialize itself, journald's file descriptors are transitioned
over, mdmon jumps through hoops. Remember you want to get rid of open files
and executables and have to reexec everything, if you transition from the
initramfs root to the real root, and also from the real root to the shutdown
root.

We really don't want the IPC mechanism to be in a flux state. All tools have
to fallback to a non-standard mechanism in that case.

If I have to pull in a dbus daemon in the initramfs, we still have the
chicken and egg problem for PID 1 talking to the logging daemon and starting
dbus.
systemd cannot talk to journald via dbus unless dbus-daemon is started, dbus
cannot log anything on startup, if journald is not running, etc...

dbus-daemon would have to transition to the real root, and from the real root
to the shutdown root, without losing state.

Of course this can all be done, but it would involve fallback mechanisms,
which we want to get rid off. Hopefully, you don't suggest to merge dbus with
PID 1. Also with a daemon, you will lose the points mentioned in the cover mail
:

 * Security: The peers which communicate do not have to trust each
   other, as the only trustworthy component in the game is the kernel
   which adds metadata and ensures that all data passed as payload is
   either copied or sealed, so that the receiver can parse the data
   without having to protect against changing memory while parsing
   buffers. Also, all the data transfer is controlled by the kernel,
   so that LSMs can track and control what is going on, without
   involving userspace. Because of the LSM issue, security people are
   much happier with this model than the current scheme of having to
   hook into dbus to mediate things.

 * Being in the kernel closes a lot of races which can't be fixed with
   the current userspace solutions.  For example, with kdbus, there is a
   way a client can disconnect from a bus, but do so only if no further
   messages present in its queue, which is crucial for implementing
   race-free exit-on-idle services

 * Eavesdropping on the kernel level, so privileged users can hook into
   the message stream without hacking support for that into their
   userspace processes

 * A number of smaller benefits: for example kdbus learned a way to peek
   full messages without dequeing them, which is really useful for
   logging metadata when handling bus-activation requests.

I don't care, if the kdbus speedup is only marginal.

In my ideal world, there is a standard IPC mechanism from the beginning to
the end, which does not rely on any process running (except the kernel) and
which is used by _all_ tools, be it a system daemon providing information and
interfaces about device assembly or network setup tools or end user desktop
processes.

dbus _is_ such an easy, flexible standard IPC mechanism. Of course, you 

Re: [GIT PULL] kdbus for 4.1-rc1

[Martin Steigerwald]

Am Mittwoch, 29. April 2015, 11:03:41 schrieb Theodore Ts'o:
 On Wed, Apr 29, 2015 at 04:53:53PM +0200, Harald Hoyer wrote:
  Sure, I can write one binary to rule them all, pull out all the code
  from all tools I need, but for me an IPC mechanism sounds a lot
  better. And it should be _one_ common IPC mechanism and not a
  plethora of them. It should feel like an operating system and not
  like a bunch of thrown together software, which is glued together
  with some magic shell scripts.
 
 And so requiring wireshark (and X?) in initramfs to debug problems
 once dbus is introduced is better?
 
 I would think shell scripts are *easier* to debug when things go
 wrong, especially in a minimal environment such as an initial ram
 disk.  Having had to debug problems in a distro initramfs when trying
 to help a customer bring up a FC boot disk long ago in another life,
 I'm certain I would rather debug problems while on site at a
 classified machine room[1] using shell scripts, and trying to debug
 dbus is something that would be infinitely worse.

Later in boot process I have seen some debugging issues with a systemd 
governed userspace:

Bug 1213778 - drops into emergency mode without any error message if it 
cannot find a filesystem in /etc/fstab
https://bugzilla.redhat.com/show_bug.cgi?id=1213778

Bug 1213781 - does not start ssh service if a filesystem in /etc/fstab 
cannot be mounted
https://bugzilla.redhat.com/show_bug.cgi?id=1213781

With shell scripts I had error messages for these, here I had to browse 
the output of journalctl -xb to find out.

And then what do I do if dbus communication is not working for some 
reason? Will I then be able to use journalctl at all?

Not that proper error reporting can´t be added in systemd and the 
dependency handling can´t be fixed towards some more sanity like for 
example no /boot mount, no worries, I still start that ssh service for 
you, but currently for me this is a clear and heavy regression when I 
compare this with sysvinit boot behavior.

I do fix things when being dropped to initramfs shell, I added some script 
snippet for supporting BTRFS RAID 1 while it wasn´t yet supported in 
Debian (dunno if it is for booting in Jessie, but as my bug reports have 
not been closed yet, maybe bot), I still want to be able to do these kinds 
of things.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[David Lang]

On Wed, 29 Apr 2015, Martin Steigerwald wrote:


Am Mittwoch, 29. April 2015, 14:47:53 schrieb Harald Hoyer:

We really don't want the IPC mechanism to be in a flux state. All tools
have to fallback to a non-standard mechanism in that case.

If I have to pull in a dbus daemon in the initramfs, we still have the
chicken and egg problem for PID 1 talking to the logging daemon and
starting dbus.
systemd cannot talk to journald via dbus unless dbus-daemon is started,
dbus cannot log anything on startup, if journald is not running, etc...


Do I get this right that it is basically a userspace *design* decision
that you use as a reason to have kdbus inside the kernel?

Is it really necessary to use DBUS for talking to journald? And does it
really matter that much if any message before starting up dbus do not
appear in the log? /proc/kmsg is a ring buffer, it can still be copied over
later.


I've been getting the early boot messages in my logs for decades (assuming the 
system doesn't fail before the syslog daemon is started). It sometimes has 
required setting a larger than default ringbuffer in the kernel, but that's easy 
enough to do.


David Lang
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[John Stoffel]

 Harald == Harald Hoyer har...@redhat.com writes:

Harald On 29.04.2015 15:33, Richard Weinberger wrote:
 It depends how you define beginning. To me an initramfs is a *very* minimal
 tool to prepare the rootfs and nothing more (no udev, no systemd, no
 mini distro).
 If the initramfs fails to do its job it can print to the console like
 the kernel does if it fails
 at a very early stage.
 

Harald Your solution might work for your small personal needs, but
Harald not for our customers.

Arguing that your needs outweight mine because you have customers
ain't gonna fly... I don't care about your customers and why should I?
I'm not getting any money from them.  Nor do I make any money from
Linux kernel though as an IT person, I support Linux all day long.  Do
my requirements get listened to as well?

If your customers wnat this feature, you're more than welcome to fork
the kernel and support it yourself.  Oh wait... Redhat does that
already.  So what's the problem?   Just put it into RHEL (which I use
I admit, along with Debian/Mint) and be done with it.

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[David Lang]

On Wed, 29 Apr 2015, Andy Lutomirski wrote:


On Wed, Apr 29, 2015 at 12:30 PM, Austin S Hemmelgarn
ahferro...@gmail.com wrote:

On 2015-04-29 14:54, Andy Lutomirski wrote:


On Apr 29, 2015 5:48 AM, Harald Hoyer har...@redhat.com wrote:



  * Being in the kernel closes a lot of races which can't be fixed with
the current userspace solutions.  For example, with kdbus, there is a
way a client can disconnect from a bus, but do so only if no further
messages present in its queue, which is crucial for implementing
race-free exit-on-idle services



This can be implemented in userspace.

Client to dbus daemon: may I exit now?
Dbus daemon to client: yes (and no more messages) or no


Depending on how this is implemented, there would be a potential issue if a
message arrived for the client after the daemon told it it could exit, but
before it finished shutdown, in which case the message might get lost.



Then implement it the right way?  The client sends some kind of
sequence number with its request.


so any app in the system can prevent any other app from exiting/restarting by 
just sending it the equivalent of a ping over dbus?


preventing an app from exiting because there are unhandled messages doesn't mean 
that those messages are going to be handled, just that they will get read and 
dropped on the floor by an app trying to exit. Sometimes you will just end up 
with a hung app that can't process messages and needs to be restarted, but can't 
be restarted because there are pending messages.


The problem with guaranteed delivery messages is that things _will_ go wrong 
that will cause the messages to not be received and processed. At that point you 
have the choice of loosing some messages or freezing your entire system (you can 
buffer them for some time, but eventually you will run out of buffer space)


We see this all the time in the logging world, people configure their systems 
for reliable delivery of log messages to a remote machine, then when that remote 
machine goes down and can't receive messages (or a network issue blocks the 
traffic), the sending machine blocks and causes an outage.


Being too strict about guaranteeing delivery just doesn't work. You must have a 
mechanism to abort and throw away unprocessed messages. If this means 
disconnecting the receiver so that there are no missing messages to the 
receiver, that's a valid choice. But preventing a receiver from exiting because 
it hasn't processed a message is not a valid choice.


David Lang
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Michal Hocko]

On Mon 27-04-15 13:11:03, Andy Lutomirski wrote:
 [resent without HTML]
 
 On Apr 27, 2015 5:46 AM, Michal Hocko mho...@suse.cz wrote:
 
  On Wed 22-04-15 12:36:12, Andy Lutomirski wrote:
[...]
   The receiver gets to mmap the buffer.  I'm not sure what protection they 
   get.
 
  OK, so I've checked the code. kdbus_pool_new sets up a shmem file
  (unlinked) so not visible externally. The consumer will get it via mmap
  on the endpoint file by kdbus_pool_mmap and it refuses VM_WRITE and
  clears VM_MAYWRITE. The receiver even doesn't have access to the shmem
  file directly.
 
  It is ugly that kdbus_pool_mmap replaces the original vm_file and make
  it point to the shmem file. I am not sure whether this is safe all the
  time and it would deserve a big fat comment. On the other hand, it seems
  some drivers are doing this already (e.g. dma_buf_mmap).
 
 What happens to map_files in proc?  It seems unlikely that CRIU would
 ever work on dma_buf, but this could be a problem for CRIU with kdbus.

I am not familiar with CRIU and likewise with map_files directory. I've
actually heard about it for the first time.
[looking...]

So proc_map_files_readdir will iterate all VMAs including the one backed
by the buffer and it will see it's vm_file which will point to the shmem
file AFAICS. So it doesn't seem like CRIU would care because all parties
on the buffer would see the same inode. Whether that is really enough, I
dunno.
 
   The thing I'm worried about is that the receiver might deliberately
   avoid faulting in a bunch of pages and instead wait for the producer
   to touch them, causing pages that logically belong to the receiver to
   be charged to the producer instead.
 
  Hmm, now that I am looking into the code it seems you are right. E.g.
  kdbus_cmd_send runs in the context of the sender AFAIU. This gets down
  to kdbus_pool_slice_copy_iovec which does vfs_iter_write and this
  is where we get to charge the memory. AFAIU the terminology all the
  receivers will share the same shmem file when mmaping the endpoint.
 
  This, however, doesn't seem to be exploitable to hide memory charges
  because the receiver cannot make the buffer writable. A nasty process
  with a small memcg limit could still pre-fault the memory before any
  writer gets sends a message and slow the whole endpoint traffic. But
  that wouldn't be a completely new thing because processes might hammer
  on memory even without memcg... It is just that this would be kind of
  easier with memcg.
  If that is the concern then the buffer should be pre-charged at the time
  when it is created.
 
 The attach I had in mind was that the nasty process with a small memcg
 creates one or many of these and doesn't pre-fault it. Then a sender
 (systemd?) sends messages and they get charged, possibly once for each
 copy sent, to the root memcg.

Dunno but I suspect that systemd will not talk to random endpoints. Or
can those endpoints be registered on a systembus by an untrusted task?
Bus vs. endpoint relation is still not entirely clear to me.

But even if that was possible I fail to see how the small memcg plays
any role when the task doesn't control the shmem buffer directly but it
only has a read only mapping of it.

 So kdbus should probably pre-charge the creator of the pool.

Yes, as I've said above.
-- 
Michal Hocko
SUSE Labs
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Steven Rostedt]

On Wed, Apr 29, 2015 at 12:26:59PM -0400, John Stoffel wrote:
 
 If your customers wnat this feature, you're more than welcome to fork
 the kernel and support it yourself.  Oh wait... Redhat does that
 already.  So what's the problem?   Just put it into RHEL (which I use
 I admit, along with Debian/Mint) and be done with it.

Red Hat tries very hard to push things upstream. It's policy is to not keep
things for themselves, but always work with the community. That way, everyone
benefits. Ideally, we should come up with a solution that works for all.

-- Steve

--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[John Stoffel]

 Steven == Steven Rostedt rost...@goodmis.org writes:

Steven On Wed, Apr 29, 2015 at 12:26:59PM -0400, John Stoffel wrote:
 
 If your customers wnat this feature, you're more than welcome to fork
 the kernel and support it yourself.  Oh wait... Redhat does that
 already.  So what's the problem?   Just put it into RHEL (which I use
 I admit, along with Debian/Mint) and be done with it.

Steven Red Hat tries very hard to push things upstream. It's policy
Steven is to not keep things for themselves, but always work with the
Steven community. That way, everyone benefits. Ideally, we should
Steven come up with a solution that works for all.

Yeah, I agree they have been good.  I'm just reacting to the off the
cuff comment of my customers need it which isn't a justification for
this feature, esp when it hasn't been shown to be needed in the
kernel.  

We went through alot of this with tux the in-kernel httpd server, and
pushing other stuff out to user-space over the years.  Why this needs
to come in isn't clear.  Or why not just a small part needing to come
in with the rest in userspace.   

John
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Austin S Hemmelgarn]

On 2015-04-29 14:54, Andy Lutomirski wrote:

On Apr 29, 2015 5:48 AM, Harald Hoyer har...@redhat.com wrote:


  * Being in the kernel closes a lot of races which can't be fixed with
the current userspace solutions.  For example, with kdbus, there is a
way a client can disconnect from a bus, but do so only if no further
messages present in its queue, which is crucial for implementing
race-free exit-on-idle services


This can be implemented in userspace.

Client to dbus daemon: may I exit now?
Dbus daemon to client: yes (and no more messages) or no

Depending on how this is implemented, there would be a potential issue 
if a message arrived for the client after the daemon told it it could 
exit, but before it finished shutdown, in which case the message might 
get lost.




smime.p7s
Description: S/MIME Cryptographic Signature

Re: [GIT PULL] kdbus for 4.1-rc1

[Martin Steigerwald]

Am Mittwoch, 29. April 2015, 13:39:42 schrieb Steven Rostedt:
 On Wed, Apr 29, 2015 at 12:26:59PM -0400, John Stoffel wrote:
  If your customers wnat this feature, you're more than welcome to fork
  the kernel and support it yourself.  Oh wait... Redhat does that
  already.  So what's the problem?   Just put it into RHEL (which I use
  I admit, along with Debian/Mint) and be done with it.
 
 Red Hat tries very hard to push things upstream. It's policy is to not
 keep things for themselves, but always work with the community. That
 way, everyone benefits. Ideally, we should come up with a solution that
 works for all.

I think work with the community is a two-way process.

Two way as in actually really *listening* to feedback instead of trying to 
push things as much as possible, believing to be *right* about things. I 
honestly dislike the I know it better than you, go away kind of attitude 
I have seen again and again. Here, in systemd-devel (where I unsubscribed 
again as I saw no use in continuing the discussion there) and even in 
Debian mailinglists and bug reports.

Wherever I look what I call the systemd approach triggers intense 
polarity and resistance. With that I do not say there is something wrong 
about it, yet, I ask myself, why is that? And my best answer I came up 
with up to now comes back to how proponents of the new, different, not 
necessary better or worse, way, treat feedback.

There I found to some extent: taking the feedback into account and 
actually adressing it. Especially when the feedback fitted into the new way 
of doing things.

Yet I also found:

- I know it better than you, go away.

- Please only stick to pure technical reasons as in Whats wrong with 
the code? disregarding any concerns about the *concept* and about 
different oppinions about whether kdbus code actually really belongs into 
the kernel

- Ignoring it


So I still say the issues are not purely technical. So I think a purely 
technical as in what´s wrong with the code? approach will not address 
the core of this discussion and the strong resistance against merging 
kdbus into the kernel.

It sometimes appears to me like childs arguing about whether to paint 
their favorite toy red or green. I think a healthy approach might be to 
agree to disagree and work from there. That would at least break the I am 
right, No, I am right cycle.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Andy Lutomirski]

On Wed, Apr 29, 2015 at 12:30 PM, Austin S Hemmelgarn
ahferro...@gmail.com wrote:
 On 2015-04-29 14:54, Andy Lutomirski wrote:

 On Apr 29, 2015 5:48 AM, Harald Hoyer har...@redhat.com wrote:


   * Being in the kernel closes a lot of races which can't be fixed with
 the current userspace solutions.  For example, with kdbus, there is a
 way a client can disconnect from a bus, but do so only if no further
 messages present in its queue, which is crucial for implementing
 race-free exit-on-idle services


 This can be implemented in userspace.

 Client to dbus daemon: may I exit now?
 Dbus daemon to client: yes (and no more messages) or no

 Depending on how this is implemented, there would be a potential issue if a
 message arrived for the client after the daemon told it it could exit, but
 before it finished shutdown, in which case the message might get lost.


Then implement it the right way?  The client sends some kind of
sequence number with its request.

--Andy
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Andy Lutomirski]

On Apr 29, 2015 5:48 AM, Harald Hoyer har...@redhat.com wrote:
 Of course this can all be done, but it would involve fallback mechanisms,
 which we want to get rid off. Hopefully, you don't suggest to merge dbus with
 PID 1. Also with a daemon, you will lose the points mentioned in the cover 
 mail
 :

  * Security: The peers which communicate do not have to trust each
other, as the only trustworthy component in the game is the kernel
which adds metadata and ensures that all data passed as payload is
either copied or sealed, so that the receiver can parse the data
without having to protect against changing memory while parsing
buffers. Also, all the data transfer is controlled by the kernel,
so that LSMs can track and control what is going on, without
involving userspace. Because of the LSM issue, security people are
much happier with this model than the current scheme of having to
hook into dbus to mediate things.

Other security people prefer code to stay out of the kernel when possible.

Also, this metadata argument is still invalid.  Sockets can be
improved for this purpose.


  * Being in the kernel closes a lot of races which can't be fixed with
the current userspace solutions.  For example, with kdbus, there is a
way a client can disconnect from a bus, but do so only if no further
messages present in its queue, which is crucial for implementing
race-free exit-on-idle services

This can be implemented in userspace.

Client to dbus daemon: may I exit now?
Dbus daemon to client: yes (and no more messages) or no


  * Eavesdropping on the kernel level, so privileged users can hook into
the message stream without hacking support for that into their
userspace processes


Why would it be a hack in userspace but not a hack in the kernel?

  * A number of smaller benefits: for example kdbus learned a way to peek
full messages without dequeing them, which is really useful for
logging metadata when handling bus-activation requests.

MSG_PEEK?


 I don't care, if the kdbus speedup is only marginal.

 In my ideal world, there is a standard IPC mechanism from the beginning to
 the end, which does not rely on any process running (except the kernel) and
 which is used by _all_ tools, be it a system daemon providing information and
 interfaces about device assembly or network setup tools or end user desktop
 processes.

 dbus _is_ such an easy, flexible standard IPC mechanism. Of course, you can
 invent the wheel again (NIH, we know better) and wait and see, if that
 works out. Until then the whole common IPC problem is unresolved and Linux
 distributions are just a collection of random software with no common
 interoperability and home grown interfaces.

I don't think anyone is suggesting throwing out dbus.


 [1] transitioning mdmon is one of the critical parts for an IMSM raid array.
 Also running an executable from a disk, which the executable is monitoring,
 and which stops functioning, if the executable is not responding is insane.


That sounds like an excellent reason to just keep running the
initramfs copy, and I don't see why IPC has anything to do with where
the executable comes from.

--Andy
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Stephen Smalley]

On 04/29/2015 11:18 AM, Simon McVittie wrote:
 On 29/04/15 14:35, Stephen Smalley wrote:
 It is also interesting that kdbus allows impersonation of any
 credential, including security label, by privileged clients, where
 privileged simply means it either has CAP_IPC_OWNER or owns (euid
 matches uid) the bus.
 
 FWIW, this particular feature is *not* one of those that are necessary
 for feature parity with dbus-daemon. There's no API for making
 dbus-daemon fake its clients' credentials; if you can ptrace it, then
 you can of course subvert it arbitrarily, but nothing less hackish than
 that is currently offered.

Then I'd be inclined to drop it from kdbus unless some compelling use
case exists, and even then, I don't believe that CAP_IPC_OWNER or
bus-owner uid match is sufficient even for forging credentials other
than the security label.  For socket credentials passing, for example,
the kernel checks CAP_SYS_ADMIN for pid forging, CAP_SETUID for uid
forging, and CAP_SETGID for gid forging.  And I don't believe we support
any form of forging of the security label on socket credentials.

 For feature parity with dbus-daemon, the fact that
 eavesdropping/monitoring *exists* is necessary (it's a widely used
 developer/sysadmin feature) but the precise mechanics of how you get it
 are not necessarily set in stone. In particular, if you think kdbus'
 definition of are you privileged? may be too broad, that seems a valid
 question to be asking.
 
 In traditional D-Bus, individual users can normally eavesdrop/monitor on
 their own session buses (which are not a security boundary, unless
 specially reconfigured), and this is a useful property; on non-LSM
 systems without special configuration, each user should ideally be able
 to monitor their own kdbus user bus, too.
 
 The system bus *is* a security boundary, and administrative privileges
 should be required to eavesdrop on it. At a high level, someone with
 full root privileges should be able to eavesdrop, and ordinary users
 should not; there are various possible criteria for distinguishing
 between those two extremes, and I have no opinion on whether
 CAP_IPC_OWNER is the most appropriate cutoff point.
 
 In dbus-daemon, LSMs with integration code in dbus-daemon have the
 opportunity to mediate eavesdropping specially. SELinux does not
 currently do this (as far as I can see), but AppArmor does, so
 AppArmor-confined processes are not normally allowed to eavesdrop on the
 session bus (even though the same user's unconfined processes may). That
 seems like one of the obvious places for an LSM hook in kdbus.

Yes, we would want to control this in SELinux; I suspect that either the
eavesdropping functionality did not exist in dbus-daemon at the time of
the original dbus-daemon SELinux integration or it was an oversight.

 Having eavesdropping be unobservable means that applications cannot
 change their behaviour while they are being watched, either maliciously
 (to hide from investigation) or accidentally (bugs that only happen when
 not being debugged are the hardest to fix). dbus-daemon's traditional
 implementation of eavesdropping has had side-effects in the past, which
 is undesirable, and is addressed by the new monitoring interface in
 version 1.9. kdbus' version of eavesdropping is quite similar to the new
 monitoring interface.
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Martin Steigerwald]

Am Mittwoch, 29. April 2015, 17:22:08 schrieb Harald Hoyer:
 On 29.04.2015 17:17, Austin S Hemmelgarn wrote:
  On 2015-04-29 11:07, Harald Hoyer wrote:
  Most of the stuff does not work without udev and something like
  systemd. 
  That's funny, apparently the initramfs images I've been using for
  multiple months now on server systems at work which don't have
  systemd, udev, or dbus, and do LVM/RAID assembly, network
  configuration, crypto devices, multipath, many different filesystems,
  and a number of other oddball configurations due to the insanity that
  is the software I have to deal with from our company, don't work.  I
  wonder how my systems are booting successfully 100% of the time then?
 Then you should probably open source your initramfs, so we can all
 benefit from it and use it for all distributions.

Do you really think that the tooling will make that much of a difference?

I think there will always be cases where a initramfs will not work until 
adapted to it. And then its nice, to be able to do things like this:

merkaba:~ cat /etc/initramfs-tools/scripts/local-top/btrfs
#!/bin/sh

PREREQ=lvm
prereqs()
{
echo $PREREQ
}

case $1 in
prereqs)
prereqs
exit 0;
esac

. /scripts/functions

log_begin_msg Initializing BTRFS RAID-1.

modprobe btrfs
vgchange -ay
btrfs device scan

log_end_msg


How would I add support for some configuration that a systemd or purely 
dracut + udev based initramfs does not support *yet*, on my own?

Yes, one can argue, why doesn´t Debian support it already, but heck, I can 
do it myself and report a bug about it, without having to fire up a C 
compiler in order to fix things. I may be able to do this myself, but at a 
much higher cost in time.

Above thing works so long already that I even often forgot about it.

That said, if I still get a chance to execute a script at some time, a 
dracut based initramfs may just be totally fine with it, but I want this 
possibility and a shell to fix things up myself it they go wrong. And while 
I do not get the need for having systemd in the initramfs at all, I might 
be fine with it, if I can fix things up myself in case of problems.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: [GIT PULL] kdbus for 4.1-rc1

[Paul Moore]

On Fri, Apr 24, 2015 at 4:08 AM, Karol Lewandowski
karol.k.lewandow...@gmail.com wrote:
 On Thu, Apr 23, 2015 at 09:30:13PM +0200, Greg Kroah-Hartman wrote:
 On Thu, Apr 23, 2015 at 01:42:25PM -0400, Stephen Smalley wrote:
  On 04/23/2015 01:16 PM, Greg Kroah-Hartman wrote:
   The binder developers at Samsung have stated that the implementation we
   have here works for their model as well, so I guess that is some kind of
   verification it's not entirely tied to D-Bus.  They have plans on
   dropping the existing binder kernel code and using the kdbus code
   instead when it is merged.
 
  Where do things stand wrt LSM hooks for kdbus?  I don't see any security
  hook calls in the kdbus tree except for the purpose of metadata
  collection of process security labels.  But nothing for enforcing MAC
  over kdbus IPC.  binder has a set of security hooks for that purpose, so
  it would be a regression wrt MAC enforcement to switch from binder to
  kdbus without equivalent checking there.

 There was a set of LSM hooks proposed for kdbus posted by Karol
 Lewandowsk last October, and it also included SELinux and Smack patches.
 They were going to be refreshed based on the latest code changes, but I
 haven't seen them posted, or I can't seem to find them in my limited
 email archive.

 We have been waiting for right moment with these. :-)

 Karol, what's the status of them?

 I have handed patchset over to Paul Osmialowski who started rework it for v4
 relatively recently.  I think it shouldn't be that hard to post updated 
 version...

 Paul?

Different Paul here, but very interested in the LSM and SELinux hooks
for obvious reasons; at a bare minimum please CC the LSM list on the
kdbus hooks, and preferably the SELinux list as well.  The initial
SELinux hooks I threw together were just a rough first pass, we (the
LSM and SELinux folks) need to have a better discussion about how to
provide the necessary access controls for kdbus ... preferably before
it finds its way into a released kernel.

-- 
paul moore
www.paul-moore.com
--
To unsubscribe from this list: send the line unsubscribe linux-kernel in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/