Hi Thomas, Thanks a lot for the detailed feedback, this is exactly the kind of guidance I was hoping for.
> We already have a driver for a RasPi-based USB display that someone made > in their spare time. So being a hobbyist project is not a problem per se. Good to know, thanks for clarifying that. > 2d primitives are likely not very useful for DRM. The canonical > reference of why is at [1]. The tl;dr is that there's no standard API, > and GPU-CPU transfers and setup costs are too high to make it > significantly faster than software rendering. > > Various people have proposed to add some form of 2d pipeline to DRM, but > nothing concrete has ever emerged. > > Conceptually, DRM doesn't really render anything. It composes the screen > from already-rendered buffers. Rendering to these buffers is mostly done > by Mesa drivers with some help from DRM's kernel drivers. If you want > hardware rendering, you'd need memory management for off-screen > rendering that Mesa can use independently from display output. Therefore > these simple draw and clear primitives aren't that useful. You need to > design a full rendering pipeline instead. > > [1] https://blog.ffwll.ch/2018/08/no-2d-in-drm.html Sima's blog post that you linked makes the reasoning much clearer than what I'd pieced together on my own. I understand this means the current fixed-function ioctls are really only a placeholder, not something to build on further as is. > Using a single ioctl per command will kill performance. So, if anything, > you'd want the command-buffer model. To make it fast for 2d primitives, > you'd likely have to model it like a 3d pipeline: have all 2d graphics > buffers in the display memory already and submit a large batch of > rendering commands that generate the entire screen at once. That's a helpful concrete direction, thank you. I'll look into what a command-buffer submission model would need to look like for this. > Rule of thumb is that you need a working user-space side for ioctls. > Mesa would be the premier target for 3d. > > I'm not much involved in Mesa, but I think Mesa is quickly moving > towards programmable pipelines. Getting drivers for fixed-function > hardware merged might be hard. > > How complicated is it to model a stream processor (i.e. GPU core) in VHDL? I think it's doable at a small scope. Some memory to hold uploaded bytecode, a minimal CPU core (I'd probably base the ISA on a reduced RISC-V subset, mainly to get an existing compiler toolchain for free) that processes it and writes the result into the framebuffer. That part sounds genuinely fun to build, and I'm actually looking forward to it, though it's a fair amount of work and I'll need to find the time for it. For throughput I'd design the core with multiple instances in mind from the start, e.g. 4 cores each responsible for a quarter of the screen, and scale that further once the basic version works. I'd still get a single instance working first before parallelizing it. > Can you use a PCI device for that and let the kernel do all the work? > See [2] for how to get a PCI device id. > > [2] https://www.qemu.org/docs/master/specs/pci-ids.html I want to make sure I understand this correctly before I start on it. My real hardware (DE10-Standard) stays exactly as it is, still using platform_driver via devicetree, completely unchanged. Only the QEMU device would change, from the current fixed-address platform device to a proper virtual PCI device (with a reserved vendor/device ID and BARs for VRAM/MMIO). On the driver side, that would mean adding pci_driver support alongside the existing platform_driver, so the exact same driver would run on both, just with two different probe functions depending on whether it's found via devicetree (real hardware) or via PCI enumeration (QEMU). Is that right? > That's indeed a good thing to have in hardware. > > I mentioned that the 2d/3d rendering is probably complicated to get > done. If I may suggest an alternative, you could implement additional > features of the mode-setting pipeline. Besides the primary plane that > your hardware already supports, you could add a cursor plane. Or you > could add overlay planes for displaying YUV formats (i.e., video > frames). Or you could implement existing DRM properties, such as > scaling, background colors, or HDR. These features are already > supported by user space. Your device would be usable immediately. This is a great suggestion, thank you. A hardware cursor plane actually fits something I'd already been considering for the hardware side, so that'll be my next bigger hardware update. Scaling looks approachable too, and I like that it wouldn't need extra memory on the hardware side, so I'm adding that to the list as well. Background color I think I can already cover with the existing CLEAR command. YUV overlay support is more future work for me, my current memory layout would need a bigger restructuring first to fit that in given how limited memory is on this hardware, but the idea itself sounds appealing since the VHDL side looks fairly contained. HDR isn't realistic here either way, my test display can't do it and I'd expect to hit memory limits before anything else. > Glad to hear it didn't work. drm_simple_display_pipe is obsolete and on > its way out. Please don't use it. Good to know, I'll drop that idea entirely then. Given all this, my plan for v2 is to drop the ioctl-based UAPI entirely (removing the custom ioctls means there's no UAPI to keep stable, which takes the pressure off getting the command-buffer design right immediately), address the style/review comments from Uwe and from your code review on patch 2, switch the QEMU test path from the fixed-address platform device to a proper PCI device as discussed above, and possibly add background color support since that doesn't require any hardware changes. The cursor plane, scaling, and the stream processor work will be separate follow-ups once the hardware side catches up. Thanks again for taking the time, this gives me a lot to work with. Best regards, Leander
