Re: Layering JavaFX onto an external rendering context
On Sun, 7 Mar 2021 at 15:36, Mark Raynsford wrote: > The basic primitive > that would be required is "transfer this image to this other image". > You'd need to expose that operation in way that would work for every > possible pair of rendering APIs ... The complexity of handling that would > need to be pushed onto > each person trying to use the API. Surely that is the point? I'm also not sure what you mean by "pair of rendering APIs", or why you'd want to try and support every possible pairing internally in JavaFX? > JavaFX would effectively need to > know when it was safe to use an image that was being transferred in, > without actually being able to know where the image was coming from or > who to talk to about it. The only methods it _could_ use would involve > CPU-side fences, which would then have performance implications. CPU-side fences may be required, yes. The problem you outline is already the problem we have, and have had to hack around, with the PixelBuffer API. That already lacks a way to tell JavaFX that the backing memory is no longer safe to use, or for JavaFX to communicate back that the memory is safe to reclaim - it's uni-directional. Somehow, that really needs to be made bi-directional, such that either side can take ownership and release the data as necessary. Solve that issue and you're possibly on the way to solving the same safe / unsafe concerns raised here? So, yes, potentially performance implications over complete unfettered access, no doubt. > > > Traditional stateful APIs like > > > OpenGL make it far too easy for external code to screw up the > > > rendering context anyway, so exposing JavaFX's own context would > > > be a bad idea. > > > > I remain of the view that this shouldn't really be that concerning?! > > It's a bug in the library if it does so. > > It's definitely concerning. Have you worked with OpenGL much? Yes, quite a lot. I mean "concerning" specifically to the JavaFX project, not to the library developer making use of the ability. I realise that it's a fine line to draw between the amount of access you provide, the amount you hinder future evolution, the amount you make certain use cases unviable, and the amount you allow users to shoot themselves in the foot. Best wishes, Neil -- Neil C Smith Codelerity Ltd. www.codelerity.com Codelerity Ltd. is a company registered in England and Wales Registered company number : 12063669 Registered office address : Office 4 219 Kensington High Street, Kensington, London, England, W8 6BD
Re: Layering JavaFX onto an external rendering context
On 2021-03-07T12:22:11 + Neil C Smith wrote: > Hi, > > A few comments from my perspective here, mainly coming from > frustration at the number of times I've had to tell clients or users > that JavaFX isn't a viable option for them .. > > On Sat, 6 Mar 2021 at 12:22, Mark Raynsford wrote: > > there is a > > combinatorial explosion of possibilities with regards to which > > rendering API the _user_ might be using, and which rendering API > > JavaFX might be using (if any!). > > I agree that's a difficult one, but also surely not an insurmountable > one?! There are precedents around Java APIs that provide querying and > access to optional support on a platform by platform basis... I think it's insurmountable due to the complexity of the semantics of the thing (the GPU) that's being abstracted over. The basic primitive that would be required is "transfer this image to this other image". You'd need to expose that operation in way that would work for every possible pair of rendering APIs, including getting all of the synchronization right that lower level APIs like Vulkan and DX12 require. The complexity of handling that would need to be pushed onto each person trying to use the API. JavaFX would effectively need to know when it was safe to use an image that was being transferred in, without actually being able to know where the image was coming from or who to talk to about it. The only methods it _could_ use would involve CPU-side fences, which would then have performance implications. To clarify a bit: Let's say we lived in an ideal world and JavaFX had a Vulkan backend, and contained code that was willing to cooperate with the user's Vulkan-based rendering code. The user could tell JavaFX "here's the image to use for this node, but don't use it until this semaphore becomes available". This is generally highly efficient as a semaphore (in Vulkan terms) is a GPU-side primitive that just controls parallelism across different queue instances on the GPU, with no CPU-side interaction at all. We have no way to get to that ideal world whilst JavaFX and the user's code have no common API over which to communicate. Everything would have to go via CPU-based abstractions. > > Traditional stateful APIs like > > OpenGL make it far too easy for external code to screw up the > > rendering context anyway, so exposing JavaFX's own context would > > be a bad idea. > > I remain of the view that this shouldn't really be that concerning?! > It's a bug in the library if it does so. It's definitely concerning. Have you worked with OpenGL much? If you're *lucky* you'll get a crash, because at least then you'll see the exact location where the mistake occured. What you'll typically get instead is corrupted rendering, a black screen, or just bad performance as the driver desperately tries to make sense of what you asked it to do, usually in completely unrelated parts of the rendering process. It's the prime reason I jumped ship to Vulkan the second it came out. :) > > JavaFX _does_ have a software renderer. What if we could have JavaFX > > work with entirely software rendering into an offscreen BufferedImage? > > Why BufferedImage? What about a reuse / extension / parallel of > PixelBuffer? Yes indeed. I said BufferedImage just because I assumed that would be more familiar. I'd not personally care where the output went as long as I could get the raw bytes out of it in order to upload to the GPU. -- Mark Raynsford | https://www.io7m.com
Re: Layering JavaFX onto an external rendering context
Hi, If I got you right you now changed your idea to render JavaFX into an PixelBuffer and integrate that into a scene renderer with Vulkan, ... . This exactly how the integration into SWT and Swing is done, the internal API for that is found in "com.sun.javafx.embed". Before we implemented DriftFX we integrated a JavaFX-Scene into our application exactly this way. For that purpose we hacked FXCanvas and published the changed code at our customers github account [1]. You see there at [2] how one fills the buffer from an offscreen scene. My gut feeling is that getting a "com.sun.javafx.embed"-API is much more likely than an DriftFX-Like-API but that just my personal opinion. Anyways our/my focus still is DriftFX and I/we are quite confident that what we have today in DriftFX is a good API where we today support JavaFX-8 to JavaFX-15 (I haven't tested 16 but I have not seen any changes going into it that would require us to change our code) Tom [1] https://github.com/eclipsesource/FXCanvas/blob/master/FXCanvas/src/javafx/embed/swt/FXCanvas.java [2] https://github.com/eclipsesource/FXCanvas/blob/master/FXCanvas/src/javafx/embed/swt/FXCanvas.java#L631 Am 07.03.21 um 16:35 schrieb Mark Raynsford: On 2021-03-07T12:22:11 + Neil C Smith wrote: Hi, A few comments from my perspective here, mainly coming from frustration at the number of times I've had to tell clients or users that JavaFX isn't a viable option for them .. On Sat, 6 Mar 2021 at 12:22, Mark Raynsford wrote: there is a combinatorial explosion of possibilities with regards to which rendering API the _user_ might be using, and which rendering API JavaFX might be using (if any!). I agree that's a difficult one, but also surely not an insurmountable one?! There are precedents around Java APIs that provide querying and access to optional support on a platform by platform basis... I think it's insurmountable due to the complexity of the semantics of the thing (the GPU) that's being abstracted over. The basic primitive that would be required is "transfer this image to this other image". You'd need to expose that operation in way that would work for every possible pair of rendering APIs, including getting all of the synchronization right that lower level APIs like Vulkan and DX12 require. The complexity of handling that would need to be pushed onto each person trying to use the API. JavaFX would effectively need to know when it was safe to use an image that was being transferred in, without actually being able to know where the image was coming from or who to talk to about it. The only methods it _could_ use would involve CPU-side fences, which would then have performance implications. To clarify a bit: Let's say we lived in an ideal world and JavaFX had a Vulkan backend, and contained code that was willing to cooperate with the user's Vulkan-based rendering code. The user could tell JavaFX "here's the image to use for this node, but don't use it until this semaphore becomes available". This is generally highly efficient as a semaphore (in Vulkan terms) is a GPU-side primitive that just controls parallelism across different queue instances on the GPU, with no CPU-side interaction at all. We have no way to get to that ideal world whilst JavaFX and the user's code have no common API over which to communicate. Everything would have to go via CPU-based abstractions. Traditional stateful APIs like OpenGL make it far too easy for external code to screw up the rendering context anyway, so exposing JavaFX's own context would be a bad idea. I remain of the view that this shouldn't really be that concerning?! It's a bug in the library if it does so. It's definitely concerning. Have you worked with OpenGL much? If you're *lucky* you'll get a crash, because at least then you'll see the exact location where the mistake occured. What you'll typically get instead is corrupted rendering, a black screen, or just bad performance as the driver desperately tries to make sense of what you asked it to do, usually in completely unrelated parts of the rendering process. It's the prime reason I jumped ship to Vulkan the second it came out. :) JavaFX _does_ have a software renderer. What if we could have JavaFX work with entirely software rendering into an offscreen BufferedImage? Why BufferedImage? What about a reuse / extension / parallel of PixelBuffer? Yes indeed. I said BufferedImage just because I assumed that would be more familiar. I'd not personally care where the output went as long as I could get the raw bytes out of it in order to upload to the GPU.
Re: Layering JavaFX onto an external rendering context
Why don't you just give it a try? I'd start with the native PixelBuffer option first because it is relatively simple and the performance is sufficient for most use cases. There is, of course, an overhead caused by the GPU -> CPU -> GPU memory transfer but it is not so bad if done right. I've been able to interface a JavaFX application with the external VLC video player that way. I could watch 4K videos without a problem and without stressing the CPU too much. I do not even have a dedicated GPU in my Mac Mini - just the internal Intel graphic. The nice thing about this approach is that the maximum overhead is constant. It depends only on your hardware and the size of your rendering area but not on the complexity of your rendering. Another advantage is that this approach is fully integrated into JavaFX. This means you can do anything with the rendering you can do with a JavaFX ImageView. E.g., drawing something on top of the rendering via JavaFX costs you nothing. Even double-buffering is possible if you know how to do that. And if this is not enough for you, there still is DriftFX which avoids this GPU -> CPU -> GPU memory transfer but is a bit more ambitious to handle. So I would keep that in mind as an option if necessary. Michael Am 07.03.21 um 13:22 schrieb Neil C Smith: Hi, A few comments from my perspective here, mainly coming from frustration at the number of times I've had to tell clients or users that JavaFX isn't a viable option for them .. On Sat, 6 Mar 2021 at 12:22, Mark Raynsford wrote: there is a combinatorial explosion of possibilities with regards to which rendering API the _user_ might be using, and which rendering API JavaFX might be using (if any!). I agree that's a difficult one, but also surely not an insurmountable one?! There are precedents around Java APIs that provide querying and access to optional support on a platform by platform basis. A library integrating JavaFX with some other platform / renderer specific API is always going to have to handle the situation by either failing gracefully or falling back to a less performant alternative. Traditional stateful APIs like OpenGL make it far too easy for external code to screw up the rendering context anyway, so exposing JavaFX's own context would be a bad idea. I remain of the view that this shouldn't really be that concerning?! It's a bug in the library if it does so. The current PixelBuffer API already allows enough access for a library to *really* screw up the rendering context, by bringing the whole thing crashing down. But, concurrency issues aside, adding that access has been a real boon. With access comes responsibility. JavaFX _does_ have a software renderer. What if we could have JavaFX work with entirely software rendering into an offscreen BufferedImage? Why BufferedImage? What about a reuse / extension / parallel of PixelBuffer? That could allow a range of implementation and the potential for using renderers that can output without a visible window to do this while at least keeping all pixel data off heap? Best wishes, Neil
Re: Layering JavaFX onto an external rendering context
Hi, A few comments from my perspective here, mainly coming from frustration at the number of times I've had to tell clients or users that JavaFX isn't a viable option for them .. On Sat, 6 Mar 2021 at 12:22, Mark Raynsford wrote: > there is a > combinatorial explosion of possibilities with regards to which > rendering API the _user_ might be using, and which rendering API > JavaFX might be using (if any!). I agree that's a difficult one, but also surely not an insurmountable one?! There are precedents around Java APIs that provide querying and access to optional support on a platform by platform basis. A library integrating JavaFX with some other platform / renderer specific API is always going to have to handle the situation by either failing gracefully or falling back to a less performant alternative. > Traditional stateful APIs like > OpenGL make it far too easy for external code to screw up the > rendering context anyway, so exposing JavaFX's own context would > be a bad idea. I remain of the view that this shouldn't really be that concerning?! It's a bug in the library if it does so. The current PixelBuffer API already allows enough access for a library to *really* screw up the rendering context, by bringing the whole thing crashing down. But, concurrency issues aside, adding that access has been a real boon. With access comes responsibility. > JavaFX _does_ have a software renderer. What if we could have JavaFX > work with entirely software rendering into an offscreen BufferedImage? Why BufferedImage? What about a reuse / extension / parallel of PixelBuffer? That could allow a range of implementation and the potential for using renderers that can output without a visible window to do this while at least keeping all pixel data off heap? Best wishes, Neil -- Neil C Smith Codelerity Ltd. www.codelerity.com Codelerity Ltd. is a company registered in England and Wales Registered company number : 12063669 Registered office address : Office 4 219 Kensington High Street, Kensington, London, England, W8 6BD
Re: Layering JavaFX onto an external rendering context
On 2021-02-15T13:40:43 + Mark Raynsford wrote: > Hello! > > I'd like to use JavaFX for the UI of an application that will > involve rendering using an existing Vulkan-based renderer. For the sake > of example, assume that the application looks and behaves a bit like > the Unreal Engine 4 editing tools Apologies for the delay in responding (I've read and digested every reply). So... Having read everything, I'm reasonably sure that the ability to do this in the "traditional" way - that is, having a node in the JavaFX scene graph that displays the contents of a GPU-side image/texture - isn't going to be happening any time soon. There are multiple intersecting issues that make this intractably difficult to achieve: 1. There are software boundaries; JavaFX doesn't expose the internal rendering context it uses and, even if it did, there is a combinatorial explosion of possibilities with regards to which rendering API the _user_ might be using, and which rendering API JavaFX might be using (if any!). Traditional stateful APIs like OpenGL make it far too easy for external code to screw up the rendering context anyway, so exposing JavaFX's own context would be a bad idea. 2. There are hardware and concurrency issues; Let's assume that JavaFX is using OpenGL or DirectX internally. Let's also assume that the user is using some kind of API that explicitly provides for managing concurrency with primitives such as fences, semaphores, etc. The user would have to somehow magically arrange for an image transfer on the GPU into an image that JavaFX knew about, and would have to somehow set up all of the right memory barriers and notify JavaFX when the transfer was completed. This is likely to be intensely error-prone at the very least. The thought of implementing this directly brings me out in a cold sweat. So how about another approach entirely? In the past, I've rendered UI elements using entirely software rendering and then uploaded the software-rendered UI as a texture to be overlaid on top of the 3D rendered view. I could obviously continue to do this, but maintaining entire software-rendered UI toolkits myself is a burden, and it would be great if I could use an existing known-good toolkit instead like JavaFX. JavaFX _does_ have a software renderer. What if we could have JavaFX work with entirely software rendering into an offscreen BufferedImage? To make this work, we'd need the following: 1. We'd need to be able to initialize JavaFX in a manner that _didn't_ require JavaFX to "own" the primary stage. That is, right now JavaFX will invoke a user's Application subclass with a Stage that JavaFX created by itself. This won't work for most rendering engines, as typically the programmer has to do a fair bit of work to open a graphics context with the correct settings. Consider the typical code needed to open a window over GLFW with Vulkan... https://github.com/io7m/jcoronado/blob/develop/com.io7m.jcoronado.examples/src/main/java/com/io7m/jcoronado/examples/HelloVulkan.java 2. We'd need to be able to force JavaFX to operate entirely using Java2D software rendering. This shouldn't be a problem at all. 3. We'd need to be able to make JavaFX render to an offscreen BufferedImage. It would be the responsibility of the programmer to submit keyboard and mouse events to JavaFX manually, because JavaFX would no longer "own" the primary application window. I believe the first half of this is already doable in the sense that the Node class allows for taking snapshots of a scene; these snapshots could be uploaded to the GPU as textures. The second half I believe is not exposed in any sense; JavaFX opens its own windows, and consumes input events from those windows directly. I wonder if it would be possible to, for example, add some kind of WindowFactory API to JavaFX so that, when JavaFX wants to create a Window, there's the option to delegate window creation to the user. The created windows might not even be operating system windows (for window-in-window UI systems)[0]. As long as the returned Window instance acted like a Window in terms of returning dimensions, producing the right input events... Would JavaFX even know that it wasn't dealing with an operating system window? [0] https://softcamel.com/wp-content/uploads/2019/03/Transport-Tycoon-Deluxe-2.png -- Mark Raynsford | https://www.io7m.com
Re: Layering JavaFX onto an external rendering context
I don't know if this is useful or not, but I've pretty succesfully combined a JavaFX UI with the MPV video player (also VLC), without resorting to any kind of frame copying. It basically involves finding the HWND id (or Window id on Linux) of a JavaFX Stage, then telling MPV / VLC to render directly to that window. A 2nd (transparent where needed) window is then used to render the JavaFX content. A top-level Stage is created, for which we find out the HWND. Here we allow MPV to render its content. The top-level Stage also has a child stage, which tracks the size and location of the main stage. This stage is transparent and contains a JavaFX Scene that can be given a (partially) transparent background made to show the content of the main stage. Child stages automatically appear on top of their parent stage. Here's a screenshot of how that looks, where the control panel, the timer, clock and pause indicator are JavaFX and the background is MPV: https://github.com/hjohn/MediaSystem-v2/blob/master/screenshot-5.jpg Although this works pretty well, there are some limitations. It may not work as well on Linux or Mac, as I rarely test this solution there. Secondly, you cannot do any kind of special effects on the MPV content (like a blur or something). --John On 15/02/2021 14:40, Mark Raynsford wrote: Hello! I'd like to use JavaFX for the UI of an application that will involve rendering using an existing Vulkan-based renderer. For the sake of example, assume that the application looks and behaves a bit like the Unreal Engine 4 editing tools. Here's an example of those: https://www.youtube.com/watch?v=2UowdJetXwA My understanding right now is that there isn't direct support in JavaFX for building this kind of application, and the primary reason for this is that there's a sort of conceptual wrestling match for control of a platform-specific rendering context here. For example: * A JavaFX application will tell JavaFX to open a new window, and the JavaFX implementation will do all of the OS-windowing-system-specific things to achieve this, and will also set up a system-specific rendering context depending on what the underlying platform is (OpenGL, DirectX, Metal, etc). JavaFX then translates input such as mouse and keyboard events from OS-specific types to the platform-independent JavaFX types so that the application can process them. * A typical Vulkan application will ask something analogous to the GLFW library to open a new window, and set up a rendering context. The GLFW library then translates input such as mouse and keyboard events from OS-specific types to generic GLFW event types, and the Vulkan application (probably) translates these into its own application-specific event types for processing. Obviously, in a given application, we're limited to having either one of these things happen, but realistically not both. The current approach (as of JavaFX 14) seems to be to use the PixelBuffer API in order to provide a CPU-side bridge between JavaFX and whatever rendering system is being used for external 3D rendering. In other words, this is the expected setup: 1. A JavaFX application will tell JavaFX to open a new window, and JavaFX will do all of the system-specific work required as described previously. 2. The application will then tell a library such as GLFW to create an _offscreen_ rendering context, perhaps configuring Vulkan or OpenGL. 3. The application, at the end of each frame, copies the contents of the offscreen rendering context's framebuffer into a PixelBuffer instance to be displayed inside a JavaFX UI. This, as far as I know, works correctly. The main complaint with this is that it pays a pretty heavy price: There's one framebuffer-sized copy operation from the GPU to the CPU (in order to read the required pixels), and then another framebuffer-sized copy operation back from the CPU to the GPU (either when writing into the PixelBuffer, or when JavaFX renders the contents of that PixelBuffer to the screen). My understanding is that it's essentially necessary to do these two rather expensive copying operations merely because JavaFX can't and won't expose the underlying rendering context it uses for its own UI rendering, and it also can't be expected to talk to whatever other rendering system the application might be using. The problem is essentially "we have these two systems both using the GPU, but they don't know each other and therefore we can't write code to get memory from one to the other without going via the CPU". Is this an accurate picture of the situation? As someone working exclusively with Vulkan, I can arrange to have the GPU copy the framebuffer into host-visible (not necessarily host-resident, but host _visible_) memory at the end of each frame. It's a little sad to have to actually copy that memory over the PCI bus just to immediately copy it back again, though. Is
Re: Layering JavaFX onto an external rendering context
There was a thread around obsolete rendering code (1) around removal of Prisim rendering and related api layers architectural elements later on. Is this relevant here? Eric (1) https://mail.openjdk.java.net/pipermail/openjfx-dev/2021-January/028581.html On Wed, Feb 17, 2021 at 8:11 AM Herve Girod wrote: > There is also something that we should be aware of. The external graphic > context is a fragile thing. In our case, for example with OpenGL, it was > very easy to create problems with the Java app which try to paint things on > the context. Which can lead to crashes or artefacts. > > Le mer. 17 févr. 2021 à 12:06, Neil C Smith a écrit > : > > > On Tue, 16 Feb 2021 at 23:33, Michael Strauß > > wrote: > > > The main problem with this idea is that there is no universally > available > > > hardware rendering backend in JavaFX. There's OpenGL on Linux and > macOS, > > > Direct3D on Windows, and potentially a software renderer on all > > platforms. > > > > How is that a problem? Not all platforms support PosixFilePermissions > > either. I used that io -> nio2 comparison because of that similar > > choice of lowest denominator abstraction as opposed to an API for > > querying capabilities and exposing them if available. Most, if not > > all, of the use cases here are about interaction with libraries using > > native components that are either not universally available or provide > > platform-specific alternatives too? > > > > Incidentally, does the OpenGL renderer not work on Windows at all, or > > just not get used by default? Can't remember. > > > > > It is generally not safe to expose the OpenGL rendering context > > > that is used internally by JavaFX, because users might inadvertently > > change > > > the GL state machine. > > > > Why is that actually a problem? Surely caveat emptor has to apply > > here? And potentially access can be managed within scopes that > > require permissions, push/pop state, etc if required? > > > > Best wishes, > > > > Neil > > > > -- > > Neil C Smith > > Codelerity Ltd. > > www.codelerity.com > > > > Codelerity Ltd. is a company registered in England and Wales > > Registered company number : 12063669 > > Registered office address : Office 4 219 Kensington High Street, > > Kensington, London, England, W8 6BD > > > -- Eric Bresie ebre...@gmail.com
Re: Layering JavaFX onto an external rendering context
There is also something that we should be aware of. The external graphic context is a fragile thing. In our case, for example with OpenGL, it was very easy to create problems with the Java app which try to paint things on the context. Which can lead to crashes or artefacts. Le mer. 17 févr. 2021 à 12:06, Neil C Smith a écrit : > On Tue, 16 Feb 2021 at 23:33, Michael Strauß > wrote: > > The main problem with this idea is that there is no universally available > > hardware rendering backend in JavaFX. There's OpenGL on Linux and macOS, > > Direct3D on Windows, and potentially a software renderer on all > platforms. > > How is that a problem? Not all platforms support PosixFilePermissions > either. I used that io -> nio2 comparison because of that similar > choice of lowest denominator abstraction as opposed to an API for > querying capabilities and exposing them if available. Most, if not > all, of the use cases here are about interaction with libraries using > native components that are either not universally available or provide > platform-specific alternatives too? > > Incidentally, does the OpenGL renderer not work on Windows at all, or > just not get used by default? Can't remember. > > > It is generally not safe to expose the OpenGL rendering context > > that is used internally by JavaFX, because users might inadvertently > change > > the GL state machine. > > Why is that actually a problem? Surely caveat emptor has to apply > here? And potentially access can be managed within scopes that > require permissions, push/pop state, etc if required? > > Best wishes, > > Neil > > -- > Neil C Smith > Codelerity Ltd. > www.codelerity.com > > Codelerity Ltd. is a company registered in England and Wales > Registered company number : 12063669 > Registered office address : Office 4 219 Kensington High Street, > Kensington, London, England, W8 6BD >
Re: Layering JavaFX onto an external rendering context
On Tue, 16 Feb 2021 at 23:33, Michael Strauß wrote: > The main problem with this idea is that there is no universally available > hardware rendering backend in JavaFX. There's OpenGL on Linux and macOS, > Direct3D on Windows, and potentially a software renderer on all platforms. How is that a problem? Not all platforms support PosixFilePermissions either. I used that io -> nio2 comparison because of that similar choice of lowest denominator abstraction as opposed to an API for querying capabilities and exposing them if available. Most, if not all, of the use cases here are about interaction with libraries using native components that are either not universally available or provide platform-specific alternatives too? Incidentally, does the OpenGL renderer not work on Windows at all, or just not get used by default? Can't remember. > It is generally not safe to expose the OpenGL rendering context > that is used internally by JavaFX, because users might inadvertently change > the GL state machine. Why is that actually a problem? Surely caveat emptor has to apply here? And potentially access can be managed within scopes that require permissions, push/pop state, etc if required? Best wishes, Neil -- Neil C Smith Codelerity Ltd. www.codelerity.com Codelerity Ltd. is a company registered in England and Wales Registered company number : 12063669 Registered office address : Office 4 219 Kensington High Street, Kensington, London, England, W8 6BD
Re: Layering JavaFX onto an external rendering context
On Tue, 16 Feb 2021 at 20:41, Mark Raynsford wrote: > I've used > GStreamer outside of Java quite a bit for aggregating feeds from > network cameras... Yes, just working with a client doing exactly that - and the UI won't be using JavaFX there, in fact they're still using Swing/AWT because it's easier to attach window overlays there. > In your case, is it that you want to consume one or > more video streams from GStreamer inside a JavaFX UI? We'd be pretty > close in terms of requirements with regards to not copying across > CPU/GPU boundaries, and requiring low latency. :) Yes, that would be part of it. Although I can also see potential cases for putting aspects of a JavaFX display back into a GStreamer pipeline, for streaming, overlays, etc. > It's why I said in my original email that I > was resigned to the fact that a copy was probably going to have to > happen somewhere. Yes, the PixelBuffer issue we faced is slightly different there. In fact, in theory we don't need an extra copy on the software side. Taking GStreamer as a specific example, although it's not the only native API that works in this way, you have a bunch of buffers allocated externally from a pool. On the JavaFX side, you want to atomically swap what buffer is in use, and release the old one back to the native library, which might end up freeing the memory. Just removing all references on the JavaFX event thread isn't (wasn't?) atomic - the renderer thread might still try and access the memory to upload the data later, which might segfault. Best wishes, Neil -- Neil C Smith Codelerity Ltd. www.codelerity.com Codelerity Ltd. is a company registered in England and Wales Registered company number : 12063669 Registered office address : Office 4 219 Kensington High Street, Kensington, London, England, W8 6BD
Re: Layering JavaFX onto an external rendering context
On 2021-02-16T16:09:04 + Neil C Smith wrote: > > I agree with you, and have certain similar requirements, like being > able to allow GStreamer and JavaFX to share GPU contexts. In fact, > was bugging Johan about this in the chat around his FOSDEM talk, and > promised to follow up here, so might as well pop my head above the > parapet. :-) I do think it ultimately boils down to "provide an image to JavaFX when it asks for one, and don't allow the image to leave the GPU". I've used GStreamer outside of Java quite a bit for aggregating feeds from network cameras... In your case, is it that you want to consume one or more video streams from GStreamer inside a JavaFX UI? We'd be pretty close in terms of requirements with regards to not copying across CPU/GPU boundaries, and requiring low latency. :) > I certainly don't know what such an API should look like, but in some > ways to me parallels the differences between io file and nio2 files - > hidden by abstraction vs type-safe queryable capabilities / profiles. > In fact, given above, also something slightly akin to GStreamer's > context querying. > > Incidentally, reading your initial post about PixelBuffer reminds me > that I should also follow up on a point about concurrency issues with > that from last year. The API has (or had?) issues with accessing the > buffer from the rendering thread after it's been removed in the event > thread, which particularly with externally allocated buffers makes it > hard to safely mark as invalid to allow them to be freed. Yeah, that is a problem. It's why I said in my original email that I was resigned to the fact that a copy was probably going to have to happen somewhere. Without going into API specifics, I was sort of picturing JavaFX maintaining two images on the GPU; at any given time one is being displayed (ie. used as a texture whilst rendering a scene) and the other is being offered to the user as a destination for a GPU copy operation. It would be the responsibility of the user to copy an image in the right format into the offered buffer. There would need to be the appropriate memory barriers inserted, and I don't know how this could/would be handled given that it might be two completely different graphics APIs involved (there isn't a Vulkan JavaFX backend, for example,so in my case I'm guaranteed to be talking to something that isn't Vulkan). When an image had been copied in as required, and JavaFX had been told that this had happened, the two images would be swapped on the GPU. -- Mark Raynsford | https://www.io7m.com
Re: Layering JavaFX onto an external rendering context
The main problem with this idea is that there is no universally available hardware rendering backend in JavaFX. There's OpenGL on Linux and macOS, Direct3D on Windows, and potentially a software renderer on all platforms. One approach might be to create a new ANGLE rendering backend, which would allow you to use the OpenGL API with any kind of platform-specific API. If such a rendering backend is ever created, it might actually be sensible to deprecate the existing D3D and GLES backends. Since ANGLE also supports Vulkan, it might be unnecessary to implement a new backend for Vulkan support specifically. As far as JavaFX API is concerned, it should be relatively straightforward to create a control that can host a custom GL surface similar to D3DImage in WPF. It is generally not safe to expose the OpenGL rendering context that is used internally by JavaFX, because users might inadvertently change the GL state machine. A better approach would be to create a new shared context that can share surfaces with the main JavaFX GL context. Of course, having all of that might still not be really useful without a Java-based API for OpenGL. Perhaps JOGL could be adapted to work for this scenario? Am Di., 16. Feb. 2021 um 21:42 Uhr schrieb Mark Raynsford < org.open...@io7m.com>: > On 2021-02-16T16:09:04 + > Neil C Smith wrote: > > > > I agree with you, and have certain similar requirements, like being > > able to allow GStreamer and JavaFX to share GPU contexts. In fact, > > was bugging Johan about this in the chat around his FOSDEM talk, and > > promised to follow up here, so might as well pop my head above the > > parapet. :-) > > I do think it ultimately boils down to "provide an image to JavaFX when > it asks for one, and don't allow the image to leave the GPU". I've used > GStreamer outside of Java quite a bit for aggregating feeds from > network cameras... In your case, is it that you want to consume one or > more video streams from GStreamer inside a JavaFX UI? We'd be pretty > close in terms of requirements with regards to not copying across > CPU/GPU boundaries, and requiring low latency. :) > > > I certainly don't know what such an API should look like, but in some > > ways to me parallels the differences between io file and nio2 files - > > hidden by abstraction vs type-safe queryable capabilities / profiles. > > In fact, given above, also something slightly akin to GStreamer's > > context querying. > > > > Incidentally, reading your initial post about PixelBuffer reminds me > > that I should also follow up on a point about concurrency issues with > > that from last year. The API has (or had?) issues with accessing the > > buffer from the rendering thread after it's been removed in the event > > thread, which particularly with externally allocated buffers makes it > > hard to safely mark as invalid to allow them to be freed. > > Yeah, that is a problem. It's why I said in my original email that I > was resigned to the fact that a copy was probably going to have to > happen somewhere. Without going into API specifics, I was sort of > picturing JavaFX maintaining two images on the GPU; at any given time > one is being displayed (ie. used as a texture whilst rendering a scene) > and the other is being offered to the user as a destination for a GPU > copy operation. It would be the responsibility of the user to copy an > image in the right format into the offered buffer. There would need to > be the appropriate memory barriers inserted, and I don't know how this > could/would be handled given that it might be two completely different > graphics APIs involved (there isn't a Vulkan JavaFX backend, for > example,so in my case I'm guaranteed to be talking to something that isn't > Vulkan). When an image had been copied in as required, and JavaFX had > been told that this had happened, the two images would be swapped on the > GPU. > > -- > Mark Raynsford | https://www.io7m.com > >
Re: Layering JavaFX onto an external rendering context
On 2021-02-15T14:54:43 +0100 Hervé Girod wrote: > I did that with OpenGL some time ago. I should setup a GitHub project to show > how it can be done. > I appreciate the response, but there is a difference between "I did it" and "there's actually a good, officially supported way to do this". I've seen people do various tricks like those in DriftFX: https://github.com/eclipse-efx/efxclipse-drift/ But it seems to me like those projects shouldn't _have_ to exist. It seems like a bit of a design flaw that there isn't an efficient code path to do something (relatively) simple like this. Has anyone given any thought as to what an API like this should look like? -- Mark Raynsford | https://www.io7m.com
Re: Layering JavaFX onto an external rendering context
Hi Mark, On Tue, 16 Feb 2021 at 15:27, Mark Raynsford wrote: > But it seems to me like those projects shouldn't _have_ to exist. It > seems like a bit of a design flaw that there isn't an efficient code > path to do something (relatively) simple like this. > > Has anyone given any thought as to what an API like this should look > like? I agree with you, and have certain similar requirements, like being able to allow GStreamer and JavaFX to share GPU contexts. In fact, was bugging Johan about this in the chat around his FOSDEM talk, and promised to follow up here, so might as well pop my head above the parapet. :-) I certainly don't know what such an API should look like, but in some ways to me parallels the differences between io file and nio2 files - hidden by abstraction vs type-safe queryable capabilities / profiles. In fact, given above, also something slightly akin to GStreamer's context querying. Incidentally, reading your initial post about PixelBuffer reminds me that I should also follow up on a point about concurrency issues with that from last year. The API has (or had?) issues with accessing the buffer from the rendering thread after it's been removed in the event thread, which particularly with externally allocated buffers makes it hard to safely mark as invalid to allow them to be freed. Best wishes, Neil -- Neil C Smith Codelerity Ltd. www.codelerity.com Codelerity Ltd. is a company registered in England and Wales Registered company number : 12063669 Registered office address : Office 4 219 Kensington High Street, Kensington, London, England, W8 6BD
Layering JavaFX onto an external rendering context
Hello! I'd like to use JavaFX for the UI of an application that will involve rendering using an existing Vulkan-based renderer. For the sake of example, assume that the application looks and behaves a bit like the Unreal Engine 4 editing tools. Here's an example of those: https://www.youtube.com/watch?v=2UowdJetXwA My understanding right now is that there isn't direct support in JavaFX for building this kind of application, and the primary reason for this is that there's a sort of conceptual wrestling match for control of a platform-specific rendering context here. For example: * A JavaFX application will tell JavaFX to open a new window, and the JavaFX implementation will do all of the OS-windowing-system-specific things to achieve this, and will also set up a system-specific rendering context depending on what the underlying platform is (OpenGL, DirectX, Metal, etc). JavaFX then translates input such as mouse and keyboard events from OS-specific types to the platform-independent JavaFX types so that the application can process them. * A typical Vulkan application will ask something analogous to the GLFW library to open a new window, and set up a rendering context. The GLFW library then translates input such as mouse and keyboard events from OS-specific types to generic GLFW event types, and the Vulkan application (probably) translates these into its own application-specific event types for processing. Obviously, in a given application, we're limited to having either one of these things happen, but realistically not both. The current approach (as of JavaFX 14) seems to be to use the PixelBuffer API in order to provide a CPU-side bridge between JavaFX and whatever rendering system is being used for external 3D rendering. In other words, this is the expected setup: 1. A JavaFX application will tell JavaFX to open a new window, and JavaFX will do all of the system-specific work required as described previously. 2. The application will then tell a library such as GLFW to create an _offscreen_ rendering context, perhaps configuring Vulkan or OpenGL. 3. The application, at the end of each frame, copies the contents of the offscreen rendering context's framebuffer into a PixelBuffer instance to be displayed inside a JavaFX UI. This, as far as I know, works correctly. The main complaint with this is that it pays a pretty heavy price: There's one framebuffer-sized copy operation from the GPU to the CPU (in order to read the required pixels), and then another framebuffer-sized copy operation back from the CPU to the GPU (either when writing into the PixelBuffer, or when JavaFX renders the contents of that PixelBuffer to the screen). My understanding is that it's essentially necessary to do these two rather expensive copying operations merely because JavaFX can't and won't expose the underlying rendering context it uses for its own UI rendering, and it also can't be expected to talk to whatever other rendering system the application might be using. The problem is essentially "we have these two systems both using the GPU, but they don't know each other and therefore we can't write code to get memory from one to the other without going via the CPU". Is this an accurate picture of the situation? As someone working exclusively with Vulkan, I can arrange to have the GPU copy the framebuffer into host-visible (not necessarily host-resident, but host _visible_) memory at the end of each frame. It's a little sad to have to actually copy that memory over the PCI bus just to immediately copy it back again, though. Is there no design we could come up with that would allow for at worst a simple GPU → GPU copy? I'm resigned to the fact that a copying operation is probably going to happen _somewhere_, but it'd be nice if we could avoid a rather expensive and redundant GPU → CPU → GPU copy. -- Mark Raynsford | https://www.io7m.com
Re: Layering JavaFX onto an external rendering context
I did that with OpenGL some time ago. I should setup a GitHub project to show how it can be done. Sent from my iPhone > On Feb 15, 2021, at 14:41, Mark Raynsford wrote: > > Hello! > > I'd like to use JavaFX for the UI of an application that will > involve rendering using an existing Vulkan-based renderer. For the sake > of example, assume that the application looks and behaves a bit like > the Unreal Engine 4 editing tools. Here's an example of those: > > https://www.youtube.com/watch?v=2UowdJetXwA > > My understanding right now is that there isn't direct support in > JavaFX for building this kind of application, and the primary reason > for this is that there's a sort of conceptual wrestling match for > control of a platform-specific rendering context here. For example: > > * A JavaFX application will tell JavaFX to open a new window, >and the JavaFX implementation will do all of the >OS-windowing-system-specific things to achieve this, and will >also set up a system-specific rendering context depending on >what the underlying platform is (OpenGL, DirectX, Metal, etc). >JavaFX then translates input such as mouse and keyboard events >from OS-specific types to the platform-independent JavaFX types >so that the application can process them. > > * A typical Vulkan application will ask something analogous to >the GLFW library to open a new window, and set up a rendering >context. The GLFW library then translates input such as mouse and >keyboard events from OS-specific types to generic GLFW event >types, and the Vulkan application (probably) translates these >into its own application-specific event types for processing. > > Obviously, in a given application, we're limited to having either > one of these things happen, but realistically not both. > > The current approach (as of JavaFX 14) seems to be to use the > PixelBuffer API in order to provide a CPU-side bridge between > JavaFX and whatever rendering system is being used for external 3D > rendering. In other words, this is the expected setup: > > 1. A JavaFX application will tell JavaFX to open a new window, > and JavaFX will do all of the system-specific work required > as described previously. > > 2. The application will then tell a library such as GLFW to > create an _offscreen_ rendering context, perhaps configuring > Vulkan or OpenGL. > > 3. The application, at the end of each frame, copies the contents > of the offscreen rendering context's framebuffer into a PixelBuffer > instance to be displayed inside a JavaFX UI. > > This, as far as I know, works correctly. The main complaint with > this is that it pays a pretty heavy price: There's one framebuffer-sized > copy operation from the GPU to the CPU (in order to read the required > pixels), and then another framebuffer-sized copy operation back from > the CPU to the GPU (either when writing into the PixelBuffer, or when > JavaFX renders the contents of that PixelBuffer to the screen). > > My understanding is that it's essentially necessary to do these two > rather expensive copying operations merely because JavaFX can't and > won't expose the underlying rendering context it uses for its own UI > rendering, and it also can't be expected to talk to whatever other > rendering system the application might be using. The problem is > essentially "we have these two systems both using the GPU, but they > don't know each other and therefore we can't write code to get > memory from one to the other without going via the CPU". > > Is this an accurate picture of the situation? > > As someone working exclusively with Vulkan, I can arrange to have > the GPU copy the framebuffer into host-visible (not necessarily > host-resident, but host _visible_) memory at the end of each frame. > It's a little sad to have to actually copy that memory over the PCI bus > just to immediately copy it back again, though. Is there no design we > could come up with that would allow for at worst a simple GPU → GPU > copy? I'm resigned to the fact that a copying operation is probably > going to happen _somewhere_, but it'd be nice if we could avoid a > rather expensive and redundant GPU → CPU → GPU copy. > > -- > Mark Raynsford | https://www.io7m.com >
