Re: RFC: libei - emulated input in Wayland compositors
An update to the work that's been happening here. libei now has some xdg-desktop-portal support. It will ping the portal, get the fd back and then go on from there. For Xwayland switching to use the a portal is a one-line change, so a sandboxed X application now has XTEST requests go through the portal. The portal implementation is currently very simple and for initial negotiation only, once the fd has been exchanged, the portal has no further say in the matter. I suspect that's likely to change one way or another. Olivier has been working on the Xwayland support, after some initial testing we figured it's *not* possible to have libei support batch requests directly. This is required for xdotool which fires off all requests and exits immediately. Olivier has added event buffering so we can replay the events once the device negotiation is done. He's also added the bits so we can "identify" clients based on their X client name. There's a new part to manage Restrictions for EIS, libreis. The portal can use that on the fd to limit what the client will be able to do later. Right now the portal sets the client's name to the app-id (and that name cannot be changed by the client, this also applies to Xwayland). It's early stages otherwise, lock down of capabilities (e.g. deny keyboard caps) is mostly just sketched out, not tested. I've been filling in the libei/libeis pieces though a lot of it is still in the proof-of-concept stage. Do not run your nuclear fusion plant on this. Either way, I reckon the whole thing looks promising. Combination of branches for portal support is outlined here: https://gitlab.freedesktop.org/whot/libei/-/issues/1 Comes with free occasional crashes. Cheers, Peter On Fri, Jul 31, 2020 at 03:13:50PM +1000, Peter Hutterer wrote: > I've been working on a new approach for allowing emulated input devices in > Wayland. Or in short - how can we make xdotool and synergy work? And > eventually replace them. > > The proposal I have is a library for Emulated Input, in short libei. > https://gitlab.freedesktop.org/whot/libei/ > > libei has two parts, the client side (libei) for applications and > a server side (libeis) for the compositor. The two libraries communicate > with each other (how? doesn't matter, it's an implementation detail) to > negotiate input devices. > > The process is roughly: > - the libei client connects and says "I am org.freedesktop.SomeApplication > and I want a pointer and a keyboard device" > - the libeis server says "ok, you can have a pointer device and a keyboard > device" > - the libei client says 'move the pointer by 1/1', etc. and the server does > just that. or not, depending on context. > > There are more details, see the README in the repo and the libei.h and > libeis.h header files that describe the API. > > The sticking point here is: emulated input comes via a separate channel. > The server a) knows it's emulated input, b) knows who it is coming from and > c) has complete control over the input. > > a) is interesting because you can differ between the events internally. The > API right now is very similar to libinput's events so integrating it into a > compositor should be trivial. > > b) is somewhat handwavy if an application runs outside a sandbox - any > information will be unreliable. Flatpak gives you an app-id though and > with that we can (eventually) do things like storing the allow/deny > decisions of the user in the portal implementation. > > c) allows you to e.g. suspend the client when convenient or just ignore > certain sequences altogether. The two made-up examples are: suspend EI > during a password prompt, or allow EI from the software yubikey *only* > during a password prompt. > > Now, the next question is: how do they *start* talking to each other? > libei provides multiple backends for the initial connection negotiation. My > goal is to have this work with flatpak portals so an application running > within the sandbox can be restricted accordingly. Alternatives to this could > be public DBus interfaces, simple fd passing or (as is implemented right > now) a named unix socket. > > The aim is that a client can simply iterate through all of the options until > finds a connection. Once that's found, the actual code for emulating input is > always the same so it's trivial to implement a client that works on any > compositor that supports some backend of libeis. > The server part only needs to care about the negotiation mechanisms it > allows, i.e. GNOME will only have dbus/portal, sway will only have... dunno, > fd exchange maybe? > > Next: because we have a separate channel for emulated input we can hook up > XTEST to use libei to talk to a compositor. I have a PoC implementation for > weston and Xwayland: > https://gitlab.freedesktop.org/whot/weston/-/commits/wip/eis > https://gitlab.freedesktop.org/whot/xserver/-/commits/wip/xwayland-eis > With that xdotool can move the pointer. Note this is truly the most
Re: RFC: libei - emulated input in Wayland compositors
On Mon, Aug 03, 2020 at 02:57:11PM +1000, Peter Hutterer wrote:
> On Sat, Aug 01, 2020 at 01:42:16PM +0200, Roman Gilg wrote:
> > >
> > > 1) It exports a set of APIs under org.freedesktop.portal.* that all
> > > sandboxed applications can access.
> > >
> > > In contrast to explicitly allowed APIs (i.e. build time configured list
> > > of API to be exposed directly to inside an application sandbox by
> > > default), a portal APIs allows for applications to dynamically request
> > > access to privileged functionality, for example access to arbitrary
> > > file system locations, cameras, geo location, or screen casting.
> >
> > That's quite a long sentence. Let me dissect it a bit. What kind of
> > API are we talking about? The xdg-portal-desktop one? Who defines the
> > "build time configured list of API"? Are we talking about the "build"
> > of the Flatpak? Is it the Manifest's "finish-args" as described this
> > way in the Flatpak documentation? [1]
>
> sort-of, there's a confusion of terms. "API" stands primarily for a DBus API
> but you'll probably think of X11 and Wayland as sockets. for the purpose
> here you can consider them as APIs as well.
>
> When you build a flatpak application, you specify what that application can
> access outside the sandbox, that's your [1] link. by default, a sandboxed
> application does not have X11 or Wayland access.
>
> you can allow access to those or any API at build time, e.g.
> --talk-name=org.freedesktop.EI would allow the application to talk to a EIS
> server directly without the portal in between. These permissions are
> compiled in and listed by the flatpak itself - they're not dynamic like the
> portal.
>
> Problem with any build-time permissions is the portal has no control over
> it. This particularly shows the issue with sockets which are a complete
> side-channel. A DBus API will be called every time
> it's needed - that allows for a gatekeeper in between. A socket, once
> established, is largely outside the control of such a gatekeeper. XTEST for
> example floats along on the rest of X, you cannot allow an application to
> use X for display but disallow XTEST in a portal, support for this would
> have to be implemented in X itself. The same would be true for a Wayland
> virtual input protocol.
>
> EI is (will be) a combination of API + socket, similar to screencasting -
> negotiation is handled by the portal but once the connection is established
> the data is outside the portal's control. We could make the communication
> fully dbus-y so we can filter at any point but I don't think that would
> provide a lot of benefit.
>
> The socket approach here is acceptable because it's very *specific* for what
> it does. It only does emulated input, so giving permission to that implies
> you want that to happen. Giving permission for X (or Wayland) does not by
> default mean you want emulated input to happen through those protocols.
>
To expand on this, a side channel can be technically relatively multi
purpose, e.g. PipeWire, which is used both for screen casting access,
camera access, and eventually microphone input, sound output etc.
To make this possible, the service provider (PipeWire) needs to
integrate with the sandbox eco system some how, and PipeWire does this
by allowing two phased initialization.
This could in theory be applied to Ei sessions too.
To summarize how access management is handled when using PipeWire, there
are currently two scenarios: screen cast and camera.
For screen casting, the portal sets up screen cast PipeWire streams.
It then creates a PipeWire remote with *only* access to these streams and
nothing else. As permissions can by the client only be restricted, and
never eased, a application cannot undo the restrictions and will thus
only ever see the screen casts it opened using the opened PipeWire file
descriptor.
For camera access, the portal opens up a PipeWire remote, it sets some
metadata ("app-id", and allowed "media roles" (i.e. "Camera")), then
locks down permission to not grant access to anything at all.
It then relies on the PipeWire session manager to know that a remote is
opened by the portal, and then open up access to devices by looking at
the metadata the portal set as well as what's in the permission store.
>
> > > 2) It provides, using backends, methods for implementing user
> > > interactive permission management. For arbitrary file system access,
> > > this may involve e.g. opening a file using a file selection dialog, or
> > > for screen casting this may mean actively choosing what part of your
> > > screen should be shared.
> >
> > Right, what the basic idea of the portals is, is pretty clear to me.
> > Basically it pipes requests of sandboxed apps through an
> > authentication and permission system in case they want to interact
> > with outside their sandbox. It's more about the cases where they
> > don't. A Wayland client in a sandbox interacts via Wayland protocol
> > with the outside of their
Re: RFC: libei - emulated input in Wayland compositors
On Sat, Aug 01, 2020 at 01:42:16PM +0200, Roman Gilg wrote: > > > > 1) It exports a set of APIs under org.freedesktop.portal.* that all > > sandboxed applications can access. > > > > In contrast to explicitly allowed APIs (i.e. build time configured list > > of API to be exposed directly to inside an application sandbox by > > default), a portal APIs allows for applications to dynamically request > > access to privileged functionality, for example access to arbitrary > > file system locations, cameras, geo location, or screen casting. > > That's quite a long sentence. Let me dissect it a bit. What kind of > API are we talking about? The xdg-portal-desktop one? Who defines the > "build time configured list of API"? Are we talking about the "build" > of the Flatpak? Is it the Manifest's "finish-args" as described this > way in the Flatpak documentation? [1] sort-of, there's a confusion of terms. "API" stands primarily for a DBus API but you'll probably think of X11 and Wayland as sockets. for the purpose here you can consider them as APIs as well. When you build a flatpak application, you specify what that application can access outside the sandbox, that's your [1] link. by default, a sandboxed application does not have X11 or Wayland access. you can allow access to those or any API at build time, e.g. --talk-name=org.freedesktop.EI would allow the application to talk to a EIS server directly without the portal in between. These permissions are compiled in and listed by the flatpak itself - they're not dynamic like the portal. Problem with any build-time permissions is the portal has no control over it. This particularly shows the issue with sockets which are a complete side-channel. A DBus API will be called every time it's needed - that allows for a gatekeeper in between. A socket, once established, is largely outside the control of such a gatekeeper. XTEST for example floats along on the rest of X, you cannot allow an application to use X for display but disallow XTEST in a portal, support for this would have to be implemented in X itself. The same would be true for a Wayland virtual input protocol. EI is (will be) a combination of API + socket, similar to screencasting - negotiation is handled by the portal but once the connection is established the data is outside the portal's control. We could make the communication fully dbus-y so we can filter at any point but I don't think that would provide a lot of benefit. The socket approach here is acceptable because it's very *specific* for what it does. It only does emulated input, so giving permission to that implies you want that to happen. Giving permission for X (or Wayland) does not by default mean you want emulated input to happen through those protocols. > > 2) It provides, using backends, methods for implementing user > > interactive permission management. For arbitrary file system access, > > this may involve e.g. opening a file using a file selection dialog, or > > for screen casting this may mean actively choosing what part of your > > screen should be shared. > > Right, what the basic idea of the portals is, is pretty clear to me. > Basically it pipes requests of sandboxed apps through an > authentication and permission system in case they want to interact > with outside their sandbox. It's more about the cases where they > don't. A Wayland client in a sandbox interacts via Wayland protocol > with the outside of their sandbox, but that's not going through > xdg-portals, or is there a permission to block it from that? For flatpak at least, you need to explicity allow wayland access (--socket=wayland) at which point it's basically a wayland-sized hole in the sandbox. > > 3) It manages remembered access, using a common permission store[0]. For > > example if an application was permanently denied camera access, the > > portal will know about this and not query the portal backend. > > Yea, I quickly did an online search for a GUI to it and found [2] > which looks awesome. Also answers my question above: there exist > permissions to block a sandboxed app from Wayland or X11. But I assume > a user won't be asked for every new GUI app he installs if it is > allowed to show some pixels and instead the permission is by default > set to allowed. Looking back at it the GUI app should set this as a > permission in their Manifest file, which is like a default on install. It does indeed ask, or at least tell you: $ flatpak install --user org.gimp.GIMP Looking for matches… Found similar ref(s) for ‘org.gimp.GIMP’ in remote ‘flathub’ (user). Use this remote? [Y/n]: y org.gimp.GIMP permissions: ipc network x11 file access [1] dbus access [2] tags [3] [1] /tmp, host, xdg-config/GIMP, xdg-config/gtk-3.0, xdg-run/gvfs [2] org.freedesktop.FileManager1, org.gtk.vfs, org.gtk.vfs.* [3] stable ... Proceed with these changes to the user installation? [Y/n]: So the built-in permissions for gimp include x11 and
Re: RFC: libei - emulated input in Wayland compositors
On Fri, Jul 31, 2020 at 08:49:41PM +0200, Roman Gilg wrote: > On Fri, Jul 31, 2020 at 7:13 AM Peter Hutterer > wrote: > > > > I've been working on a new approach for allowing emulated input devices in > > Wayland. Or in short - how can we make xdotool and synergy work? And > > eventually replace them. > > > > The proposal I have is a library for Emulated Input, in short libei. > > https://gitlab.freedesktop.org/whot/libei/ > > We talked about it already yesterday but thanks again for this great > project. I decided to directly write some experimental integration > code based on your Weston branch for the server library in KWinFT [1] > in order to try this out as a solution for my Steam Controller issue > [2] that - I assume - motivated the creation of this library to some > extent. > > And yes, it works. :) I can move the cursor with the Steam controller > as in "Steam client -> XTEST -> patched Xwayland -> libei -> libeis -> > KWinFT" just fine. > > Am I right in assuming that the button-press event is not yet done in > libei or in the patched Xwayland version you linked? When it's > available let me know and I'll add the necessary logic for that too. I've implemented buttons and keys now but note it's an optimistic implemention. i.e. not a lot of error checking and failure handling. It'll be good enough for a PoC implementation though. As before, the two examples in tools/ show what's supported and I've updated the xwayland and weston branches too (xdotool type 'hello world' works now, imagine the possibilities!). Cheers, Peter ___ wayland-devel mailing list wayland-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/wayland-devel
Re: RFC: libei - emulated input in Wayland compositors
On Sat, Aug 1, 2020 at 12:47 AM Jonas Ådahl wrote: > > On Fri, Jul 31, 2020 at 08:49:41PM +0200, Roman Gilg wrote: > > On Fri, Jul 31, 2020 at 7:13 AM Peter Hutterer > > wrote: > > > > > > I've been working on a new approach for allowing emulated input devices in > > > Wayland. Or in short - how can we make xdotool and synergy work? And > > > eventually replace them. > > > > > > The proposal I have is a library for Emulated Input, in short libei. > > > https://gitlab.freedesktop.org/whot/libei/ > > > > We talked about it already yesterday but thanks again for this great > > project. I decided to directly write some experimental integration > > code based on your Weston branch for the server library in KWinFT [1] > > in order to try this out as a solution for my Steam Controller issue > > [2] that - I assume - motivated the creation of this library to some > > extent. > > > > And yes, it works. :) I can move the cursor with the Steam controller > > as in "Steam client -> XTEST -> patched Xwayland -> libei -> libeis -> > > KWinFT" just fine. > > > > Am I right in assuming that the button-press event is not yet done in > > libei or in the patched Xwayland version you linked? When it's > > available let me know and I'll add the necessary logic for that too. > > > > > libei has two parts, the client side (libei) for applications and > > > a server side (libeis) for the compositor. The two libraries communicate > > > with each other (how? doesn't matter, it's an implementation detail) to > > > negotiate input devices. > > > > > > The process is roughly: > > > - the libei client connects and says "I am org.freedesktop.SomeApplication > > > and I want a pointer and a keyboard device" > > > - the libeis server says "ok, you can have a pointer device and a keyboard > > > device" > > > - the libei client says 'move the pointer by 1/1', etc. and the server > > > does > > > just that. or not, depending on context. > > > > > > There are more details, see the README in the repo and the libei.h and > > > libeis.h header files that describe the API. > > > > > > The sticking point here is: emulated input comes via a separate channel. > > > The server a) knows it's emulated input, b) knows who it is coming from > > > and > > > c) has complete control over the input. > > > > > > a) is interesting because you can differ between the events internally. > > > The > > > API right now is very similar to libinput's events so integrating it into > > > a > > > compositor should be trivial. > > > > > > b) is somewhat handwavy if an application runs outside a sandbox - any > > > information will be unreliable. Flatpak gives you an app-id though and > > > with that we can (eventually) do things like storing the allow/deny > > > decisions of the user in the portal implementation. > > > > > > c) allows you to e.g. suspend the client when convenient or just ignore > > > certain sequences altogether. The two made-up examples are: suspend EI > > > during a password prompt, or allow EI from the software yubikey *only* > > > during a password prompt. > > > > > > Now, the next question is: how do they *start* talking to each other? > > > libei provides multiple backends for the initial connection negotiation. > > > My > > > goal is to have this work with flatpak portals so an application running > > > within the sandbox can be restricted accordingly. Alternatives to this > > > could > > > be public DBus interfaces, simple fd passing or (as is implemented right > > > now) a named unix socket. > > > > Wiring this somehow through portals would be important for sure. > > Xwayland as a client could either be accepted by default or if > > Olivier's Xwayland xdg-portal patches [3] land (with the additional > > portal for libei) only be accepted after the user confirmed it just > > like every other sandboxed client. > > > > That being said the envisioned permission model is still somewhat > > difficult for me to grasp. To reiterate: the access of sandboxed > > clients can be accepted or rejected by the user. But to my > > understanding that's a function of the xdg-portal itself. You said the > > compositor can filter requests too. Can it only allow libei > > connections through xdg-portals and Xwayland? What about other > > clients, how can they be distinguished from xdg-portals and Xwayland > > securely? Or is this only possible for flatpaked clients? Or is such a > > client blocked from trying to do that anyway (in other words is it > > allowed or not to connect to arbitrary sockets like the libei one)? > > > > As it is probably clear now the overall concept of xdg-portals in > > detail is still not very well understood by me. From conversations I > > had lately with other windowing system developers I believe I'm not > > the only one. > > > > Since xdg-portals become more and more important for securing our > > graphical sessions it would be great if someone with more knowledge > > about it could create some kind of article or
Re: RFC: libei - emulated input in Wayland compositors
On Fri, Jul 31, 2020 at 08:49:41PM +0200, Roman Gilg wrote: > On Fri, Jul 31, 2020 at 7:13 AM Peter Hutterer > wrote: > > > > I've been working on a new approach for allowing emulated input devices in > > Wayland. Or in short - how can we make xdotool and synergy work? And > > eventually replace them. > > > > The proposal I have is a library for Emulated Input, in short libei. > > https://gitlab.freedesktop.org/whot/libei/ > > We talked about it already yesterday but thanks again for this great > project. I decided to directly write some experimental integration > code based on your Weston branch for the server library in KWinFT [1] > in order to try this out as a solution for my Steam Controller issue > [2] that - I assume - motivated the creation of this library to some > extent. > > And yes, it works. :) I can move the cursor with the Steam controller > as in "Steam client -> XTEST -> patched Xwayland -> libei -> libeis -> > KWinFT" just fine. > > Am I right in assuming that the button-press event is not yet done in > libei or in the patched Xwayland version you linked? When it's > available let me know and I'll add the necessary logic for that too. > > > libei has two parts, the client side (libei) for applications and > > a server side (libeis) for the compositor. The two libraries communicate > > with each other (how? doesn't matter, it's an implementation detail) to > > negotiate input devices. > > > > The process is roughly: > > - the libei client connects and says "I am org.freedesktop.SomeApplication > > and I want a pointer and a keyboard device" > > - the libeis server says "ok, you can have a pointer device and a keyboard > > device" > > - the libei client says 'move the pointer by 1/1', etc. and the server does > > just that. or not, depending on context. > > > > There are more details, see the README in the repo and the libei.h and > > libeis.h header files that describe the API. > > > > The sticking point here is: emulated input comes via a separate channel. > > The server a) knows it's emulated input, b) knows who it is coming from and > > c) has complete control over the input. > > > > a) is interesting because you can differ between the events internally. The > > API right now is very similar to libinput's events so integrating it into a > > compositor should be trivial. > > > > b) is somewhat handwavy if an application runs outside a sandbox - any > > information will be unreliable. Flatpak gives you an app-id though and > > with that we can (eventually) do things like storing the allow/deny > > decisions of the user in the portal implementation. > > > > c) allows you to e.g. suspend the client when convenient or just ignore > > certain sequences altogether. The two made-up examples are: suspend EI > > during a password prompt, or allow EI from the software yubikey *only* > > during a password prompt. > > > > Now, the next question is: how do they *start* talking to each other? > > libei provides multiple backends for the initial connection negotiation. My > > goal is to have this work with flatpak portals so an application running > > within the sandbox can be restricted accordingly. Alternatives to this could > > be public DBus interfaces, simple fd passing or (as is implemented right > > now) a named unix socket. > > Wiring this somehow through portals would be important for sure. > Xwayland as a client could either be accepted by default or if > Olivier's Xwayland xdg-portal patches [3] land (with the additional > portal for libei) only be accepted after the user confirmed it just > like every other sandboxed client. > > That being said the envisioned permission model is still somewhat > difficult for me to grasp. To reiterate: the access of sandboxed > clients can be accepted or rejected by the user. But to my > understanding that's a function of the xdg-portal itself. You said the > compositor can filter requests too. Can it only allow libei > connections through xdg-portals and Xwayland? What about other > clients, how can they be distinguished from xdg-portals and Xwayland > securely? Or is this only possible for flatpaked clients? Or is such a > client blocked from trying to do that anyway (in other words is it > allowed or not to connect to arbitrary sockets like the libei one)? > > As it is probably clear now the overall concept of xdg-portals in > detail is still not very well understood by me. From conversations I > had lately with other windowing system developers I believe I'm not > the only one. > > Since xdg-portals become more and more important for securing our > graphical sessions it would be great if someone with more knowledge > about it could create some kind of article or documentation about it > that looks at it from the perspective of windowing systems . How do > apps in/out of Flatpaks that display their pixels through X11, > Xwayland or Wayland directly work in respect to the sandboxed > environment provided by xdg-portals? What does this
Re: RFC: libei - emulated input in Wayland compositors
On Fri, Jul 31, 2020 at 7:13 AM Peter Hutterer wrote: > > I've been working on a new approach for allowing emulated input devices in > Wayland. Or in short - how can we make xdotool and synergy work? And > eventually replace them. > > The proposal I have is a library for Emulated Input, in short libei. > https://gitlab.freedesktop.org/whot/libei/ We talked about it already yesterday but thanks again for this great project. I decided to directly write some experimental integration code based on your Weston branch for the server library in KWinFT [1] in order to try this out as a solution for my Steam Controller issue [2] that - I assume - motivated the creation of this library to some extent. And yes, it works. :) I can move the cursor with the Steam controller as in "Steam client -> XTEST -> patched Xwayland -> libei -> libeis -> KWinFT" just fine. Am I right in assuming that the button-press event is not yet done in libei or in the patched Xwayland version you linked? When it's available let me know and I'll add the necessary logic for that too. > libei has two parts, the client side (libei) for applications and > a server side (libeis) for the compositor. The two libraries communicate > with each other (how? doesn't matter, it's an implementation detail) to > negotiate input devices. > > The process is roughly: > - the libei client connects and says "I am org.freedesktop.SomeApplication > and I want a pointer and a keyboard device" > - the libeis server says "ok, you can have a pointer device and a keyboard > device" > - the libei client says 'move the pointer by 1/1', etc. and the server does > just that. or not, depending on context. > > There are more details, see the README in the repo and the libei.h and > libeis.h header files that describe the API. > > The sticking point here is: emulated input comes via a separate channel. > The server a) knows it's emulated input, b) knows who it is coming from and > c) has complete control over the input. > > a) is interesting because you can differ between the events internally. The > API right now is very similar to libinput's events so integrating it into a > compositor should be trivial. > > b) is somewhat handwavy if an application runs outside a sandbox - any > information will be unreliable. Flatpak gives you an app-id though and > with that we can (eventually) do things like storing the allow/deny > decisions of the user in the portal implementation. > > c) allows you to e.g. suspend the client when convenient or just ignore > certain sequences altogether. The two made-up examples are: suspend EI > during a password prompt, or allow EI from the software yubikey *only* > during a password prompt. > > Now, the next question is: how do they *start* talking to each other? > libei provides multiple backends for the initial connection negotiation. My > goal is to have this work with flatpak portals so an application running > within the sandbox can be restricted accordingly. Alternatives to this could > be public DBus interfaces, simple fd passing or (as is implemented right > now) a named unix socket. Wiring this somehow through portals would be important for sure. Xwayland as a client could either be accepted by default or if Olivier's Xwayland xdg-portal patches [3] land (with the additional portal for libei) only be accepted after the user confirmed it just like every other sandboxed client. That being said the envisioned permission model is still somewhat difficult for me to grasp. To reiterate: the access of sandboxed clients can be accepted or rejected by the user. But to my understanding that's a function of the xdg-portal itself. You said the compositor can filter requests too. Can it only allow libei connections through xdg-portals and Xwayland? What about other clients, how can they be distinguished from xdg-portals and Xwayland securely? Or is this only possible for flatpaked clients? Or is such a client blocked from trying to do that anyway (in other words is it allowed or not to connect to arbitrary sockets like the libei one)? As it is probably clear now the overall concept of xdg-portals in detail is still not very well understood by me. From conversations I had lately with other windowing system developers I believe I'm not the only one. Since xdg-portals become more and more important for securing our graphical sessions it would be great if someone with more knowledge about it could create some kind of article or documentation about it that looks at it from the perspective of windowing systems . How do apps in/out of Flatpaks that display their pixels through X11, Xwayland or Wayland directly work in respect to the sandboxed environment provided by xdg-portals? What does this mean for a Wayland compositor, what does it need to do or refrain from to be on the safe side? For example some simple but lucid diagrams like the one in libei's README describing the flow around client <-> xdg-portals <-> windowing system would
Re: RFC: libei - emulated input in Wayland compositors
On Fri, Jul 31, 2020 at 12:16:55PM +0300, Pekka Paalanen wrote: > On Fri, 31 Jul 2020 15:13:50 +1000 > Peter Hutterer wrote: > > > I've been working on a new approach for allowing emulated input devices in > > Wayland. Or in short - how can we make xdotool and synergy work? And > > eventually replace them. > > > > The proposal I have is a library for Emulated Input, in short libei. > > https://gitlab.freedesktop.org/whot/libei/ > > > > libei has two parts, the client side (libei) for applications and > > a server side (libeis) for the compositor. The two libraries communicate > > with each other (how? doesn't matter, it's an implementation detail) to > > negotiate input devices. > > > > The process is roughly: > > - the libei client connects and says "I am org.freedesktop.SomeApplication > > and I want a pointer and a keyboard device" > > - the libeis server says "ok, you can have a pointer device and a keyboard > > device" > > - the libei client says 'move the pointer by 1/1', etc. and the server does > > just that. or not, depending on context. > > > > There are more details, see the README in the repo and the libei.h and > > libeis.h header files that describe the API. > > > > The sticking point here is: emulated input comes via a separate channel. > > The server a) knows it's emulated input, b) knows who it is coming from and > > c) has complete control over the input. > > > > a) is interesting because you can differ between the events internally. The > > API right now is very similar to libinput's events so integrating it into a > > compositor should be trivial. > > > > b) is somewhat handwavy if an application runs outside a sandbox - any > > information will be unreliable. Flatpak gives you an app-id though and > > with that we can (eventually) do things like storing the allow/deny > > decisions of the user in the portal implementation. > > > > c) allows you to e.g. suspend the client when convenient or just ignore > > certain sequences altogether. The two made-up examples are: suspend EI > > during a password prompt, or allow EI from the software yubikey *only* > > during a password prompt. > > > > Now, the next question is: how do they *start* talking to each other? > > libei provides multiple backends for the initial connection negotiation. My > > goal is to have this work with flatpak portals so an application running > > within the sandbox can be restricted accordingly. Alternatives to this could > > be public DBus interfaces, simple fd passing or (as is implemented right > > now) a named unix socket. > > > > The aim is that a client can simply iterate through all of the options until > > finds a connection. Once that's found, the actual code for emulating input > > is > > always the same so it's trivial to implement a client that works on any > > compositor that supports some backend of libeis. > > The server part only needs to care about the negotiation mechanisms it > > allows, i.e. GNOME will only have dbus/portal, sway will only have... dunno, > > fd exchange maybe? > > > > Next: because we have a separate channel for emulated input we can hook up > > XTEST to use libei to talk to a compositor. I have a PoC implementation for > > weston and Xwayland: > > https://gitlab.freedesktop.org/whot/weston/-/commits/wip/eis > > https://gitlab.freedesktop.org/whot/xserver/-/commits/wip/xwayland-eis > > With that xdotool can move the pointer. Note this is truly the most minimal > > code just to illustrate the point but you can fill in the blanks and do > > things like the compositor preventing XTEST or not, etc. > > > > This is all in very early stages with very little error checking so things > > will probably crash or disconnect unexpectedly. I've tried to document the > > API to make the intentions clear but there are still some very handwavy > > bits. > > > > Do let me know if you have any questions or suggestions please though. > > Hi Peter, > > this seems like a very well thought out proposal. I like it. > Particularly the "how do you connect and get authorised" is a good > compromise since there is no one solution fitting all. Thanks! > When an ei client creates a new emulated input device, how should > Wayland compositors handle that wrt. wl_seats? Add it into the main > wl_seat? Create a new wl_seat for each ei client? Is there enough > information for a compositor to decide? I think you should treat them like a physical input device, except that you know they're emulated ones. So the policy will be compositor-specific and/or client-specific (e.g. it may make sense to give the software yubikey a separate seat, if only to have a fixed layout). But the key-value storage I mentioned in the other email could be used for that. As for "is there enough information" it mostly comes down to: what do you need to make that decision? If we know that, we can figure out where to get it from. > Is it possible to "close the loop" in a Wayland/ei client, to know > which
Re: RFC: libei - emulated input in Wayland compositors
On Fri, 31 Jul 2020 15:13:50 +1000 Peter Hutterer wrote: > I've been working on a new approach for allowing emulated input devices in > Wayland. Or in short - how can we make xdotool and synergy work? And > eventually replace them. > > The proposal I have is a library for Emulated Input, in short libei. > https://gitlab.freedesktop.org/whot/libei/ > > libei has two parts, the client side (libei) for applications and > a server side (libeis) for the compositor. The two libraries communicate > with each other (how? doesn't matter, it's an implementation detail) to > negotiate input devices. > > The process is roughly: > - the libei client connects and says "I am org.freedesktop.SomeApplication > and I want a pointer and a keyboard device" > - the libeis server says "ok, you can have a pointer device and a keyboard > device" > - the libei client says 'move the pointer by 1/1', etc. and the server does > just that. or not, depending on context. > > There are more details, see the README in the repo and the libei.h and > libeis.h header files that describe the API. > > The sticking point here is: emulated input comes via a separate channel. > The server a) knows it's emulated input, b) knows who it is coming from and > c) has complete control over the input. > > a) is interesting because you can differ between the events internally. The > API right now is very similar to libinput's events so integrating it into a > compositor should be trivial. > > b) is somewhat handwavy if an application runs outside a sandbox - any > information will be unreliable. Flatpak gives you an app-id though and > with that we can (eventually) do things like storing the allow/deny > decisions of the user in the portal implementation. > > c) allows you to e.g. suspend the client when convenient or just ignore > certain sequences altogether. The two made-up examples are: suspend EI > during a password prompt, or allow EI from the software yubikey *only* > during a password prompt. > > Now, the next question is: how do they *start* talking to each other? > libei provides multiple backends for the initial connection negotiation. My > goal is to have this work with flatpak portals so an application running > within the sandbox can be restricted accordingly. Alternatives to this could > be public DBus interfaces, simple fd passing or (as is implemented right > now) a named unix socket. > > The aim is that a client can simply iterate through all of the options until > finds a connection. Once that's found, the actual code for emulating input is > always the same so it's trivial to implement a client that works on any > compositor that supports some backend of libeis. > The server part only needs to care about the negotiation mechanisms it > allows, i.e. GNOME will only have dbus/portal, sway will only have... dunno, > fd exchange maybe? > > Next: because we have a separate channel for emulated input we can hook up > XTEST to use libei to talk to a compositor. I have a PoC implementation for > weston and Xwayland: > https://gitlab.freedesktop.org/whot/weston/-/commits/wip/eis > https://gitlab.freedesktop.org/whot/xserver/-/commits/wip/xwayland-eis > With that xdotool can move the pointer. Note this is truly the most minimal > code just to illustrate the point but you can fill in the blanks and do > things like the compositor preventing XTEST or not, etc. > > This is all in very early stages with very little error checking so things > will probably crash or disconnect unexpectedly. I've tried to document the > API to make the intentions clear but there are still some very handwavy > bits. > > Do let me know if you have any questions or suggestions please though. Hi Peter, this seems like a very well thought out proposal. I like it. Particularly the "how do you connect and get authorised" is a good compromise since there is no one solution fitting all. When an ei client creates a new emulated input device, how should Wayland compositors handle that wrt. wl_seats? Add it into the main wl_seat? Create a new wl_seat for each ei client? Is there enough information for a compositor to decide? Is it possible to "close the loop" in a Wayland/ei client, to know which wl_seat is delivering the emulated input events? I'm not yet sure if that's even necessary, but I am thinking about implementing compositor input tests using ei. Do you see something missing from the ei API to work as the only API needed for writing compositor input tests? Presumably there needs to be a known policy about when the compositor creates and removes wl_seats, but maybe that can be just a internal detail shared by the compositor and its test suite. Or maybe that could be explicit through other protocol that establishes the ei connection (fd passing), so that wl_seat creation is explicit. Thanks, pq pgpbQQInSOUGX.pgp Description: OpenPGP digital signature ___ wayland-devel mailing list
Re: RFC: libei - emulated input in Wayland compositors
On Fri, Jul 31, 2020 at 11:00:53AM +0300, Vlad Zahorodnii wrote: > Howdy, > > On 7/31/20 8:13 AM, Peter Hutterer wrote: > > I've been working on a new approach for allowing emulated input devices in > > Wayland. Or in short - how can we make xdotool and synergy work? And > > eventually replace them. > > ++, it would be nice to have something that is supported by all major > compositors + working synergy on Wayland. :-) > > > The proposal I have is a library for Emulated Input, in short libei. > > https://gitlab.freedesktop.org/whot/libei/ > > > > libei has two parts, the client side (libei) for applications and > > a server side (libeis) for the compositor. The two libraries communicate > > with each other (how? doesn't matter, it's an implementation detail) to > > negotiate input devices. > > > > The process is roughly: > > - the libei client connects and says "I am org.freedesktop.SomeApplication > > and I want a pointer and a keyboard device" > > - the libeis server says "ok, you can have a pointer device and a keyboard > > device" > > - the libei client says 'move the pointer by 1/1', etc. and the server does > > just that. or not, depending on context. > > > > There are more details, see the README in the repo and the libei.h and > > libeis.h header files that describe the API. > > > > The sticking point here is: emulated input comes via a separate channel. > > The server a) knows it's emulated input, b) knows who it is coming from and > > c) has complete control over the input. > > > > a) is interesting because you can differ between the events internally. The > > API right now is very similar to libinput's events so integrating it into a > > compositor should be trivial. > > > > b) is somewhat handwavy if an application runs outside a sandbox - any > > information will be unreliable. Flatpak gives you an app-id though and > > with that we can (eventually) do things like storing the allow/deny > > decisions of the user in the portal implementation. > > > > c) allows you to e.g. suspend the client when convenient or just ignore > > certain sequences altogether. The two made-up examples are: suspend EI > > during a password prompt, or allow EI from the software yubikey *only* > > during a password prompt. > > We don't want any application be able to inject input events. How should > a program such as synergy authenticate itself? or is it completely up to > the compositor to decide what the authentication process looks like? The currently available method for application authenticaion is using portals from an application sandbox. The portal will know from what application a request to open a channel comes from without having to trust the application to tell the truth, and provide user facing ways for potentially limiting access. In other words, it'd be blocked before ever reaching the display server would so be needed. What this proposal aims is to create is (among other things) a portal (i.e. an API on org.freedesktop.portal.*) that does this. Other ways would be to make it slightly harder for applications to get access, e.g. only provide a socket name to certain ones, but that'd rely on trusting applications not to try bad things, and may have other limitations. Jonas ___ wayland-devel mailing list wayland-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/wayland-devel
Re: RFC: libei - emulated input in Wayland compositors
On Fri, Jul 31, 2020 at 11:00:53AM +0300, Vlad Zahorodnii wrote: > > c) allows you to e.g. suspend the client when convenient or just ignore > > certain sequences altogether. The two made-up examples are: suspend EI > > during a password prompt, or allow EI from the software yubikey *only* > > during a password prompt. > > We don't want any application be able to inject input events. How should > a program such as synergy authenticate itself? or is it completely up to > the compositor to decide what the authentication process looks like? yes...ish. there are two kinds of applications, the ones that run in a trusted sandbox (flatpak) and the ones that don't. For the ones outside the sandbox you can provide some information and even auto-fill it, but in the end the information is going to be untrustworthy. libei could (and probably should) pack the protocol with things like /proc/$pid/commandline but in the end you can't verify any of this. X, which basically always runs outside the sandbox could add _NET_WM_NAME but that's a voluntary property. So my current expectation is that we'll have *some* API in libei that provides for a key-value store and we'll likely agree on some keys that should go in there and classifies as untrusted but still usable data. e.g. "proxy=xwayland" or something. For flatpaks we have the app-id which is, AIUI, somewhat unfakable [1] and the portal can pass that to the libeis backend. And, again AIUI, the portal implementations already have some mechanisms to remember decisions, so once you allow com.whatever.synergy, it'll always be permitted. But either way, we're not adding anything new to that part of the process. In the end, the *decision* what's being accepted and what isn't is still outside, I think we can only provide data here. The backends will be different enough that I'm not sure we can provide some common thing here but I'm happy to be convinced otherwise. Cheers, Peter [1] installing applications that misidentify themselves is out of scope here ___ wayland-devel mailing list wayland-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/wayland-devel
Re: RFC: libei - emulated input in Wayland compositors
Howdy, On 7/31/20 8:13 AM, Peter Hutterer wrote: > I've been working on a new approach for allowing emulated input devices in > Wayland. Or in short - how can we make xdotool and synergy work? And > eventually replace them. ++, it would be nice to have something that is supported by all major compositors + working synergy on Wayland. :-) > The proposal I have is a library for Emulated Input, in short libei. > https://gitlab.freedesktop.org/whot/libei/ > > libei has two parts, the client side (libei) for applications and > a server side (libeis) for the compositor. The two libraries communicate > with each other (how? doesn't matter, it's an implementation detail) to > negotiate input devices. > > The process is roughly: > - the libei client connects and says "I am org.freedesktop.SomeApplication > and I want a pointer and a keyboard device" > - the libeis server says "ok, you can have a pointer device and a keyboard > device" > - the libei client says 'move the pointer by 1/1', etc. and the server does > just that. or not, depending on context. > > There are more details, see the README in the repo and the libei.h and > libeis.h header files that describe the API. > > The sticking point here is: emulated input comes via a separate channel. > The server a) knows it's emulated input, b) knows who it is coming from and > c) has complete control over the input. > > a) is interesting because you can differ between the events internally. The > API right now is very similar to libinput's events so integrating it into a > compositor should be trivial. > > b) is somewhat handwavy if an application runs outside a sandbox - any > information will be unreliable. Flatpak gives you an app-id though and > with that we can (eventually) do things like storing the allow/deny > decisions of the user in the portal implementation. > > c) allows you to e.g. suspend the client when convenient or just ignore > certain sequences altogether. The two made-up examples are: suspend EI > during a password prompt, or allow EI from the software yubikey *only* > during a password prompt. We don't want any application be able to inject input events. How should a program such as synergy authenticate itself? or is it completely up to the compositor to decide what the authentication process looks like? Cheers, Vlad > Now, the next question is: how do they *start* talking to each other? > libei provides multiple backends for the initial connection negotiation. My > goal is to have this work with flatpak portals so an application running > within the sandbox can be restricted accordingly. Alternatives to this could > be public DBus interfaces, simple fd passing or (as is implemented right > now) a named unix socket. > > The aim is that a client can simply iterate through all of the options until > finds a connection. Once that's found, the actual code for emulating input is > always the same so it's trivial to implement a client that works on any > compositor that supports some backend of libeis. > The server part only needs to care about the negotiation mechanisms it > allows, i.e. GNOME will only have dbus/portal, sway will only have... dunno, > fd exchange maybe? > > Next: because we have a separate channel for emulated input we can hook up > XTEST to use libei to talk to a compositor. I have a PoC implementation for > weston and Xwayland: > https://gitlab.freedesktop.org/whot/weston/-/commits/wip/eis > https://gitlab.freedesktop.org/whot/xserver/-/commits/wip/xwayland-eis > With that xdotool can move the pointer. Note this is truly the most minimal > code just to illustrate the point but you can fill in the blanks and do > things like the compositor preventing XTEST or not, etc. > > This is all in very early stages with very little error checking so things > will probably crash or disconnect unexpectedly. I've tried to document the > API to make the intentions clear but there are still some very handwavy > bits. > > Do let me know if you have any questions or suggestions please though. > > Cheers, > Peter > ___ > wayland-devel mailing list > wayland-devel@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/wayland-devel > ___ wayland-devel mailing list wayland-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/wayland-devel
RFC: libei - emulated input in Wayland compositors
I've been working on a new approach for allowing emulated input devices in Wayland. Or in short - how can we make xdotool and synergy work? And eventually replace them. The proposal I have is a library for Emulated Input, in short libei. https://gitlab.freedesktop.org/whot/libei/ libei has two parts, the client side (libei) for applications and a server side (libeis) for the compositor. The two libraries communicate with each other (how? doesn't matter, it's an implementation detail) to negotiate input devices. The process is roughly: - the libei client connects and says "I am org.freedesktop.SomeApplication and I want a pointer and a keyboard device" - the libeis server says "ok, you can have a pointer device and a keyboard device" - the libei client says 'move the pointer by 1/1', etc. and the server does just that. or not, depending on context. There are more details, see the README in the repo and the libei.h and libeis.h header files that describe the API. The sticking point here is: emulated input comes via a separate channel. The server a) knows it's emulated input, b) knows who it is coming from and c) has complete control over the input. a) is interesting because you can differ between the events internally. The API right now is very similar to libinput's events so integrating it into a compositor should be trivial. b) is somewhat handwavy if an application runs outside a sandbox - any information will be unreliable. Flatpak gives you an app-id though and with that we can (eventually) do things like storing the allow/deny decisions of the user in the portal implementation. c) allows you to e.g. suspend the client when convenient or just ignore certain sequences altogether. The two made-up examples are: suspend EI during a password prompt, or allow EI from the software yubikey *only* during a password prompt. Now, the next question is: how do they *start* talking to each other? libei provides multiple backends for the initial connection negotiation. My goal is to have this work with flatpak portals so an application running within the sandbox can be restricted accordingly. Alternatives to this could be public DBus interfaces, simple fd passing or (as is implemented right now) a named unix socket. The aim is that a client can simply iterate through all of the options until finds a connection. Once that's found, the actual code for emulating input is always the same so it's trivial to implement a client that works on any compositor that supports some backend of libeis. The server part only needs to care about the negotiation mechanisms it allows, i.e. GNOME will only have dbus/portal, sway will only have... dunno, fd exchange maybe? Next: because we have a separate channel for emulated input we can hook up XTEST to use libei to talk to a compositor. I have a PoC implementation for weston and Xwayland: https://gitlab.freedesktop.org/whot/weston/-/commits/wip/eis https://gitlab.freedesktop.org/whot/xserver/-/commits/wip/xwayland-eis With that xdotool can move the pointer. Note this is truly the most minimal code just to illustrate the point but you can fill in the blanks and do things like the compositor preventing XTEST or not, etc. This is all in very early stages with very little error checking so things will probably crash or disconnect unexpectedly. I've tried to document the API to make the intentions clear but there are still some very handwavy bits. Do let me know if you have any questions or suggestions please though. Cheers, Peter ___ wayland-devel mailing list wayland-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/wayland-devel
