Re: JavaFX Media issues
On 9/08/2013 03:48, Scott Palmer wrote: I have heard rumors of people being able to play HD video via Canvas. I have tried everything and can't come close. (Yes, I have been careful about the pixel format.) I mean, it looks like it is working for a few seconds, but then as the memory fills with the Canvas backlog it can lead to the GC using a lot more CPU, thus reducing the ability for Canvas to process its command queue even further, well it just collapses in on itself and dies. Is your app able to do *anything* else while the video is playing? The slightest delay to the rendering and that Canvas buffering bug kills the app. Not that it would matter if it could keep up, because the off-screen thing is also a deal breaker. My app is basically just a video watching system -- while video is playing (in the bottom most layer of a StackPane), it shows overlays with time / position, possibly a menu (where you can download a matching subtitle for the video and adjust settings like playback speed and brightness). These overlays are smoothly faded in / faded out just with the opacity property of another Pane that is at a higher level in the main StackPane. None of this is really CPU intensive, nor is there a huge SceneGraph to deal with (I'm guessing its smaller than 100 nodes while video playback is running). See picture here for an idea of what's going on while playback is occuring: https://github.com/hjohn/MediaSystem/blob/master/screenshot-1.jpg The Stage used is a non-transparent one -- I mention this because a transparent change performs a lot worse than a non-transparent one. Before I used the Canvas solution, I'd actually just playback video in a java.awt.Frame with a transparent JavaFX Stage on top. I'd have 2 windows, with the transparent JavaFX Stage on top that would show overlays for the video, while the other window would show the video using java.awt.Canvas, accessed directly by VLC. I do run into this issue every now and then though: https://javafx-jira.kenai.com/browse/RT-23312 It really needs to be fixed or Canvas is simply not safely usable in its current form (and IMHO never has been since its release then). Of course 25fps is well below the 60fps required for full-speed video. I suspect it is the frame rate more than the frame size that matters here. Plain old, standard definition, interlaced, 60 fields per second will probably kill most apps with the current Canvas implementation. I don't think I have aany 60fps video, if you can point me to something I can download that VLC can handle that is of the size you're using I could test with the setup I have here. With 1920x1080 HD Video, the CPU uses 8% according to Task Manager (low-end quad core xeon, about 1 year old). I'm running a standard Java VM (b99), no other tweaking, with default memory settings, 256 MB heap, Task Manager claims a working set of +/- 600 MB, but some native memory might be involved -- when playback stops, working set drops to 340 MB, so there's definitely a lot going on. It's solid though once playback starts (usually it only locks up at the start of playback, if it locks up) and can run for hours. No frame stuttering (I'd notice this immediately on a horizontal pan of one of the test videos I use). Even with a lot of other things going on on the same machine (although not by my Java process) playback stays solid -- I often have this machine running at 50-75% CPU for extended periods while enjoying a Video on the 3rd screen. I'd agree though that more than doubling the frame rate is gonna be a huge impact. I'm not certain if interlaced is going to cause any additional problems, I'm assuming that's handled by VLC and that I'd still be copying a full buffer for each frame (or do you mean 60 FPS interlaced = 120 FPS?). We need something better. I proposed having at least a JNI method so we could get native window handles from Stages but didn't get any traction either. Security was brought up as a concern, which I totally do NOT understand as the use of JNI means you are out of the Java sandbox already. If we had native window handles we might be able get our own window for rendering to at least interact nicely with the Stage. (We already did this successfully in Swing via JAWT and a heavyweight component that we paint to from native code) Getting a WindowHandle for a Stage would be great -- however, I think I actually hacked this in at one point, and then tried playing back video on a JavaFX Stage -- the video however always ended up in the foreground, obliterating any JavaFX rendering. That would need to be solved as well if it is still an issue. I've actually tried almost every video player solution that I could find (all on Windows), including DirectShow (using DSJ), GStreamer-java, MPlayer (in seperate window), VLCJ, MediaPlayer Classic Home Cinema and Xuggler. All of them except VLCJ had severe issues, ranging
Re: JavaFX Media issues
Forwarded from Kiriil Kirichenko who is getting his mailing list credentials sorted out... *From: *Felix Bembrick felix.bembr...@gmail.com mailto:felix.bembr...@gmail.com *Subject: **JavaFX Media issues* *Date: *August 8, 2013 1:31:37 PM PDT *To: *openjfx-dev@openjdk.java.net mailto:openjfx-dev@openjdk.java.net List openjfx-dev@openjdk.java.net mailto:openjfx-dev@openjdk.java.net I am having a look at JavaFX media support and have a couple of questions: 1. It seems that the only way to load media files is by specifying a source such as a file path etc. This is not going to work well for me as all of my application's content (which includes data, digital assets, media etc.) is stored in a database on the server and is loaded through an IO stream. Why doesn't Media support loading of files through a stream? That would seem like a common use case to me! This is correct. So called arbitrary input stream support is among the features for the future. 2. I am unable to locate any reference to JavaFX Media in the JavaDocs for JDK 8 which I found here: http://download.java.net/jdk8/jfxdocs/index.html Is this just a glitch? A glitch related to javadoc execution. FXMedia is still there. 3. Is buffering of media something planned for the future in JavaFX? What do you mean by buffering ? We have progressive buffer for http links. See MediaPlayer.bufferProgressTimeProperty 4. What about live streaming of media content? We support Http Live Streaming: MpegTS which contains H264 + AAC.
Re: Canvas blowing up (was Re: JavaFX Media issues)
This is a great idea. We should just enter a tweak and do it. If the area that is being cleared is larger than the current size of the canvas, we can throw away all pending draw commands. Steve On 09/08/2013 11:23 AM, Richard Bair wrote: I mean, it looks like it is working for a few seconds, but then as the memory fills with the Canvas backlog it can lead to the GC using a lot more CPU, thus reducing the ability for Canvas to process its command queue even further, well it just collapses in on itself and dies. Forking the thread. The problem with Canvas is that if you have a canvas and you scribble on it, and then scribble on it some more, and then scribble on it some more, then in order for us to get the right result in the end, we need to replay all those scribbles in order. If pulses are not happening, we still need to remember these scribbles so we can draw the right result. BUT, if you issue a command to the canvas which will cause it to clear all its contents, then we could throw away any previously buffered data. Right now the only way to do that would be a fillRect with a solid fill where the fillRect encompasses the entire canvas area, or a clearRect where the clearRect encompasses the entire canvas area. This seems like a very simple fix. GraphicsContext.clearRect and GraphicsContext.fillRect should both (under the right conditions) throw away the previously buffered commands. Then all you have to do is be sure to make one of these calls (likely just a clearRect) before each frame, and we'll never buffer more than a single frame's worth of data. We could also add a clear method which is clearRect(0, 0, w, h) to make this more foolproof, and then document it as a best practice to clear the canvas before each rendering if you intend to redraw the entire thing on each frame. If you're making use of manually operated dirty rects so that you only clear the damaged area to repaint, then we couldn't employ this technique and we'd have to buffer 'till kingdom come. So we still need a mechanism exposed in the scene graph of liveness and associated events so that when the scene is no longer live (for example, when minimized) you could stop your animation timer, but for your specific media use case this isn't as important. Richard
Re: Canvas blowing up (was Re: JavaFX Media issues)
That's okay for a quick hack. In the case of a video preview surface, I will be explicitly setting the value for every pixel from a ByteBuffer. You could save the extra step of doing a rectFill or clearRect if you knew that every pixel was about to be overwritten. It's a reasonable optimization.. but as a fix for this issue it's still only a half-fix hack. If pulses are not happening, we still need to remember these scribbles so we can draw the right result. No. If pulses are not happening you need to block or force a pulse somehow. Otherwise I don't see how having the unbounded queue is ever going to be 100% reliable. Since we are talking about painting to the Canvas surface as opposed to directly modifying the scene graph, why does the painting have to happen later when a pulse occurs? It's not like you have any other thread writing to the Canvas. Why can't the Platform thread actually *do* the scribbles,and the pulse just refreshes the portion of the Canvas that is visible on the screen? Is it some D3D/OpenGL multi-threading complication? Regards, Scott On Fri, Aug 9, 2013 at 11:43 AM, steve.x.northo...@oracle.com wrote: This is a great idea. We should just enter a tweak and do it. If the area that is being cleared is larger than the current size of the canvas, we can throw away all pending draw commands. Steve On 09/08/2013 11:23 AM, Richard Bair wrote: I mean, it looks like it is working for a few seconds, but then as the memory fills with the Canvas backlog it can lead to the GC using a lot more CPU, thus reducing the ability for Canvas to process its command queue even further, well it just collapses in on itself and dies. Forking the thread. The problem with Canvas is that if you have a canvas and you scribble on it, and then scribble on it some more, and then scribble on it some more, then in order for us to get the right result in the end, we need to replay all those scribbles in order. If pulses are not happening, we still need to remember these scribbles so we can draw the right result. BUT, if you issue a command to the canvas which will cause it to clear all its contents, then we could throw away any previously buffered data. Right now the only way to do that would be a fillRect with a solid fill where the fillRect encompasses the entire canvas area, or a clearRect where the clearRect encompasses the entire canvas area. This seems like a very simple fix. GraphicsContext.clearRect and GraphicsContext.fillRect should both (under the right conditions) throw away the previously buffered commands. Then all you have to do is be sure to make one of these calls (likely just a clearRect) before each frame, and we'll never buffer more than a single frame's worth of data. We could also add a clear method which is clearRect(0, 0, w, h) to make this more foolproof, and then document it as a best practice to clear the canvas before each rendering if you intend to redraw the entire thing on each frame. If you're making use of manually operated dirty rects so that you only clear the damaged area to repaint, then we couldn't employ this technique and we'd have to buffer 'till kingdom come. So we still need a mechanism exposed in the scene graph of liveness and associated events so that when the scene is no longer live (for example, when minimized) you could stop your animation timer, but for your specific media use case this isn't as important. Richard
Re: Canvas blowing up (was Re: JavaFX Media issues)
We would still do the fill but we could throw away any buffered commands that happened before the fill. Steve On 09/08/2013 12:16 PM, Dr. Michael Paus wrote: What would be the performance penalty for using this quick-fix? The clear/fill commands do not just clear the command buffer. They also fill the canvas area with a certain color. So in normal operation the canvas is always filled twice for each frame, isn't it? Am 09.08.13 17:23, schrieb Richard Bair: I mean, it looks like it is working for a few seconds, but then as the memory fills with the Canvas backlog it can lead to the GC using a lot more CPU, thus reducing the ability for Canvas to process its command queue even further, well it just collapses in on itself and dies. Forking the thread. The problem with Canvas is that if you have a canvas and you scribble on it, and then scribble on it some more, and then scribble on it some more, then in order for us to get the right result in the end, we need to replay all those scribbles in order. If pulses are not happening, we still need to remember these scribbles so we can draw the right result. BUT, if you issue a command to the canvas which will cause it to clear all its contents, then we could throw away any previously buffered data. Right now the only way to do that would be a fillRect with a solid fill where the fillRect encompasses the entire canvas area, or a clearRect where the clearRect encompasses the entire canvas area. This seems like a very simple fix. GraphicsContext.clearRect and GraphicsContext.fillRect should both (under the right conditions) throw away the previously buffered commands. Then all you have to do is be sure to make one of these calls (likely just a clearRect) before each frame, and we'll never buffer more than a single frame's worth of data. We could also add a clear method which is clearRect(0, 0, w, h) to make this more foolproof, and then document it as a best practice to clear the canvas before each rendering if you intend to redraw the entire thing on each frame. If you're making use of manually operated dirty rects so that you only clear the damaged area to repaint, then we couldn't employ this technique and we'd have to buffer 'till kingdom come. So we still need a mechanism exposed in the scene graph of liveness and associated events so that when the scene is no longer live (for example, when minimized) you could stop your animation timer, but for your specific media use case this isn't as important. Richard
RE: Canvas blowing up (was Re: JavaFX Media issues)
This question was recently asked on StackOverflow as well: http://stackoverflow.com/questions/18097404/how-can-i-free-canvas-memory How can I free Canvas memory? So others have been running into these kind of issues. Also the proposed clear() or empty() option only applies to Canvas correct? i.e. WritableImages don't suffer from these kind of issues and don't require such methods? Regards, John -Original Message- From: openjfx-dev-boun...@openjdk.java.net [mailto:openjfx-dev-boun...@openjdk.java.net] On Behalf Of Richard Bair Sent: Friday, August 09, 2013 9:43 AM To: Dr. Michael Paus Cc: openjfx-dev@openjdk.java.net Subject: Re: Canvas blowing up (was Re: JavaFX Media issues) What would be the performance penalty for using this quick-fix? The clear/fill commands do not just clear the command buffer. They also fill the canvas area with a certain color. So in normal operation the canvas is always filled twice for each frame, isn't it? That would be correct. Another option is to add, instead of clear() an explicit empty() method or something that would just blow away the buffer. Richard
Re: Canvas blowing up (was Re: JavaFX Media issues)
Also the proposed clear() or empty() option only applies to Canvas correct? i.e. WritableImages don't suffer from these kind of issues and don't require such methods? That is correct (WritableImage we don't provide a 2D API to use to fill the buffer, you just bash the pixels yourself however you like, so we don't have to buffer anything up). Richard
RE: JavaFX Media issues
So called arbitrary input stream support is among the features for the future. Could somebody post back a reference to the jira, I've seen the feature referred to from other jiras, but I've never been able to find the actual jira reference for this. Thanks! -Original Message- From: openjfx-dev-boun...@openjdk.java.net [mailto:openjfx-dev-boun...@openjdk.java.net] On Behalf Of Richard Bair Sent: Friday, August 09, 2013 8:35 AM To: Felix Bembrick Cc: openjfx-dev@openjdk.java.net List Subject: Re: JavaFX Media issues Forwarded from Kiriil Kirichenko who is getting his mailing list credentials sorted out... *From: *Felix Bembrick felix.bembr...@gmail.com mailto:felix.bembr...@gmail.com *Subject: **JavaFX Media issues* *Date: *August 8, 2013 1:31:37 PM PDT *To: *openjfx-dev@openjdk.java.net mailto:openjfx-dev@openjdk.java.net List openjfx-dev@openjdk.java.net mailto:openjfx-dev@openjdk.java.net I am having a look at JavaFX media support and have a couple of questions: 1. It seems that the only way to load media files is by specifying a source such as a file path etc. This is not going to work well for me as all of my application's content (which includes data, digital assets, media etc.) is stored in a database on the server and is loaded through an IO stream. Why doesn't Media support loading of files through a stream? That would seem like a common use case to me! This is correct. So called arbitrary input stream support is among the features for the future. 2. I am unable to locate any reference to JavaFX Media in the JavaDocs for JDK 8 which I found here: http://download.java.net/jdk8/jfxdocs/index.html Is this just a glitch? A glitch related to javadoc execution. FXMedia is still there. 3. Is buffering of media something planned for the future in JavaFX? What do you mean by buffering ? We have progressive buffer for http links. See MediaPlayer.bufferProgressTimeProperty 4. What about live streaming of media content? We support Http Live Streaming: MpegTS which contains H264 + AAC.
Re: JavaFX Media issues
So called arbitrary input stream support is among the features for the future. Could somebody post back a reference to the jira, I've seen the feature referred to from other jiras, but I've never been able to find the actual jira reference for this. Thanks! Here's an umbrella issue that would cover this: https://javafx-jira.kenai.com/browse/RT-14938 Unfortunately the media team is rather starved for resources these days (even more so than in the past). -DrD-
Re: Canvas blowing up (was Re: JavaFX Media issues)
On 9/08/2013 20:15, Richard Bair wrote: Also the proposed clear() or empty() option only applies to Canvas correct? i.e. WritableImages don't suffer from these kind of issues and don't require such methods? That is correct (WritableImage we don't provide a 2D API to use to fill the buffer, you just bash the pixels yourself however you like, so we don't have to buffer anything up). Hm, I didn't realize WritableImage had the same kind PixelWriter interface. For my usecase, where I just render full frames to a PixelWriter so JavaFX can display them in some fashion, why should I not use a WriteableImage instead? Looking at the docs, I see that one limitation is that the WritableImage cannot be resized after construction (something that I do use with Canvas), but I think I could work around that by just recreating the Image when its size changes... I could just wrap a class around it that does this transparently. Going to take a look if I can rip out the Canvas code and replace it with a resizable WritableImage and see what the results are for full screen video playback... --John
RE: JavaFX Media issues
JavaFX 2.2 does http live streaming: http://docs.oracle.com/javafx/2/media/overview.htm HTTP Live Streaming Support With the addition of HTTP live streaming support, you can now download the playlist file and playback video or audio segments using JavaFX Media. Media players are now able to switch to alternate streams, as specified in the playlist file and based on network conditions. For a given stream, there is a playlist file and a set of segments into which the stream is broken. The stream can be either an MP3 raw stream or an MPEG-TS containing multiplexed AAC audio and H.264 video. The stream can be played on demand when the stream is a static file or played live when the stream is actually a live feed. In both cases, the stream can adjust its bit rate and for video, its resolution can be adjusted. More info on HTTP Live Streaming: http://developer.apple.com/library/ios/#documentation/networkinginternet/conceptual/streamingmediaguide/Introduction/Introduction.html Your other questions, the JavaFX media devs can answer. -Original Message- From: openjfx-dev-boun...@openjdk.java.net [mailto:openjfx-dev-boun...@openjdk.java.net] On Behalf Of Felix Bembrick Sent: Thursday, August 08, 2013 1:32 PM To: openjfx-dev@openjdk.java.net List Subject: JavaFX Media issues I am having a look at JavaFX media support and have a couple of questions: 1. It seems that the only way to load media files is by specifying a source such as a file path etc. This is not going to work well for me as all of my application's content (which includes data, digital assets, media etc.) is stored in a database on the server and is loaded through an IO stream. Why doesn't Media support loading of files through a stream? That would seem like a common use case to me! 2. I am unable to locate any reference to JavaFX Media in the JavaDocs for JDK 8 which I found here: http://download.java.net/jdk8/jfxdocs/index.html Is this just a glitch? 3. Is buffering of media something planned for the future in JavaFX? 4. What about live streaming of media content? Thanks, Felix
Re: JavaFX Media issues
I am having a look at JavaFX media support and have a couple of questions: 1. It seems that the only way to load media files is by specifying a source such as a file path etc. This is not going to work well for me as all of my application's content (which includes data, digital assets, media etc.) is stored in a database on the server and is loaded through an IO stream. Why doesn't Media support loading of files through a stream? That would seem like a common use case to me! I will have to let the media guys chime in, I don't know why we don't support a stream. At one time Image had the same limitation but we added stream support for it, so it might just be oversight. One thing to suggest is that you can install your own URL content handlers / protocols in Java. So if your URL said myapp://nameofvideo.mov or whatever, you can install a myapp protocol handler with Java which is responsible for creating the InputStream. This way you can still had us a url and we can get back your input stream, even from a custom source such as a Database. 2. I am unable to locate any reference to JavaFX Media in the JavaDocs for JDK 8 which I found here: http://download.java.net/jdk8/jfxdocs/index.html Is this just a glitch? Probably got blown up during the gradle switch. Can you please file a bug (in the build component) 3. Is buffering of media something planned for the future in JavaFX? 4. What about live streaming of media content? Kirill should have more information here. Thanks! Richard
RE: JavaFX Media issues
One thing to suggest is that you can install your own URL content handlers / protocols in Java. That won't help in this case unless you replaced the http, file or jar protocol handlers, which would be weird. Only HTTP, FILE, and JAR URLs are supported. http://docs.oracle.com/javafx/2/api/javafx/scene/media/Media.html#Media%28java.lang.String%29 -Original Message- From: openjfx-dev-boun...@openjdk.java.net [mailto:openjfx-dev-boun...@openjdk.java.net] On Behalf Of Richard Bair Sent: Thursday, August 08, 2013 1:57 PM To: Felix Bembrick Cc: openjfx-dev@openjdk.java.net List Subject: Re: JavaFX Media issues I am having a look at JavaFX media support and have a couple of questions: 1. It seems that the only way to load media files is by specifying a source such as a file path etc. This is not going to work well for me as all of my application's content (which includes data, digital assets, media etc.) is stored in a database on the server and is loaded through an IO stream. Why doesn't Media support loading of files through a stream? That would seem like a common use case to me! I will have to let the media guys chime in, I don't know why we don't support a stream. At one time Image had the same limitation but we added stream support for it, so it might just be oversight. One thing to suggest is that you can install your own URL content handlers / protocols in Java. So if your URL said myapp://nameofvideo.mov or whatever, you can install a myapp protocol handler with Java which is responsible for creating the InputStream. This way you can still had us a url and we can get back your input stream, even from a custom source such as a Database. 2. I am unable to locate any reference to JavaFX Media in the JavaDocs for JDK 8 which I found here: http://download.java.net/jdk8/jfxdocs/index.html Is this just a glitch? Probably got blown up during the gradle switch. Can you please file a bug (in the build component) 3. Is buffering of media something planned for the future in JavaFX? 4. What about live streaming of media content? Kirill should have more information here. Thanks! Richard
Re: JavaFX Media issues
Scott, That JIRA issue doesn't make any mention of getting hooks into pixel data etc., it only refers to user defined codecs. Maybe you could open another issue to manage the wishlist of missing features such as the ones you referred to? I too would very much like to see greatly improved media support in JavaFX. Felix On 9 August 2013 10:10, Scott Palmer swpal...@gmail.com wrote: The Media APIs are mostly useless in their current state. Other than demoing that you can play a video, they don't go far enough to be of practical value. I tried to get someone to pay attention to them back in the JavaFX 1.0 days https://javafx-jira.kenai.com/browse/RT-2684 at least someone listened to the request to get H.264 support in there, but that is just a workaround. We need to be able to get our data into the media pipeline. This would allow those of us that have attempted to do a video window to have a fighting chance. Canvas can't keep up and will likely crash the app with out of memory errors. Support for drawing into a native surface (OpenGL or D3D context) has been talked about, but doesn't appear to be on the horizon yet. If we just had a hook to get the dang pixel data into the media pipeline so we could supply the next frame with whatever we want - either from any native codec via JNI, or dynamically generated from Java code, whatever... that would be just so dang useful... (to me at least) Regards, Scott On Thu, Aug 8, 2013 at 5:04 PM, Fabrizio Giudici fabrizio.giud...@tidalwave.it wrote: On Thu, 08 Aug 2013 22:57:51 +0200, Joe McGlynn joe.mcgl...@oracle.com wrote: I don't know why FX Media isn't in the FX 8 API docs, but that's clearly an error. Please file a bug on that. In the meantime, you should look at the FX 2 media docs, there isn't a lot of change from FX2 media in FX8. Buffering and streaming (HTTP Live Streaming) are both supported, as is playback from a URL. What is the strategy for codecs? I mean, now we have ImageIO (there is also JAI but it seems basically dead). ImageIO provides many image codecs and there's a SPI that can be used to support more formats. Will it be replaced by FX2 media or co-exist with it? -- Fabrizio Giudici - Java Architect @ Tidalwave s.a.s. We make Java work. Everywhere. http://tidalwave.it/fabrizio/**blog http://tidalwave.it/fabrizio/blog- fabrizio.giud...@tidalwave.it
Re: JavaFX Media issues
The Media APIs are mostly useless in their current state. Other than demoing that you can play a video, they don't go far enough to be of practical value. I tried to get someone to pay attention to them back in the JavaFX 1.0 days https://javafx-jira.kenai.com/browse/RT-2684 at least someone listened to the request to get H.264 support in there, but that is just a workaround. We need to be able to get our data into the media pipeline. This would allow those of us that have attempted to do a video window to have a fighting chance. Canvas can't keep up and will likely crash the app with out of memory errors. Support for drawing into a native surface (OpenGL or D3D context) has been talked about, but doesn't appear to be on the horizon yet. If we just had a hook to get the dang pixel data into the media pipeline so we could supply the next frame with whatever we want - either from any native codec via JNI, or dynamically generated from Java code, whatever... that would be just so dang useful... (to me at least) Regards, Scott On Thu, Aug 8, 2013 at 5:04 PM, Fabrizio Giudici fabrizio.giud...@tidalwave.it wrote: On Thu, 08 Aug 2013 22:57:51 +0200, Joe McGlynn joe.mcgl...@oracle.com wrote: I don't know why FX Media isn't in the FX 8 API docs, but that's clearly an error. Please file a bug on that. In the meantime, you should look at the FX 2 media docs, there isn't a lot of change from FX2 media in FX8. Buffering and streaming (HTTP Live Streaming) are both supported, as is playback from a URL. What is the strategy for codecs? I mean, now we have ImageIO (there is also JAI but it seems basically dead). ImageIO provides many image codecs and there's a SPI that can be used to support more formats. Will it be replaced by FX2 media or co-exist with it? -- Fabrizio Giudici - Java Architect @ Tidalwave s.a.s. We make Java work. Everywhere. http://tidalwave.it/fabrizio/**blog http://tidalwave.it/fabrizio/blog - fabrizio.giud...@tidalwave.it
Re: JavaFX Media issues
I don't want to detract from the issues around media stuff (they are significant), but if you are desperate (as I was), here's some code for doing video capture and streaming based on LTI-CIVIL and writing to JFX image to render the video: - https://code.google.com/p/jfxcamera/ It's a work around and not elegant at all, and media needs to have a lot done to be useful. I only include it here in case there is something useful you can take from it to hobble through. On Fri, Aug 9, 2013 at 10:10 AM, Scott Palmer swpal...@gmail.com wrote: The Media APIs are mostly useless in their current state. Other than demoing that you can play a video, they don't go far enough to be of practical value. I tried to get someone to pay attention to them back in the JavaFX 1.0 days https://javafx-jira.kenai.com/browse/RT-2684 at least someone listened to the request to get H.264 support in there, but that is just a workaround. We need to be able to get our data into the media pipeline. This would allow those of us that have attempted to do a video window to have a fighting chance. Canvas can't keep up and will likely crash the app with out of memory errors. Support for drawing into a native surface (OpenGL or D3D context) has been talked about, but doesn't appear to be on the horizon yet. If we just had a hook to get the dang pixel data into the media pipeline so we could supply the next frame with whatever we want - either from any native codec via JNI, or dynamically generated from Java code, whatever... that would be just so dang useful... (to me at least) Regards, Scott On Thu, Aug 8, 2013 at 5:04 PM, Fabrizio Giudici fabrizio.giud...@tidalwave.it wrote: On Thu, 08 Aug 2013 22:57:51 +0200, Joe McGlynn joe.mcgl...@oracle.com wrote: I don't know why FX Media isn't in the FX 8 API docs, but that's clearly an error. Please file a bug on that. In the meantime, you should look at the FX 2 media docs, there isn't a lot of change from FX2 media in FX8. Buffering and streaming (HTTP Live Streaming) are both supported, as is playback from a URL. What is the strategy for codecs? I mean, now we have ImageIO (there is also JAI but it seems basically dead). ImageIO provides many image codecs and there's a SPI that can be used to support more formats. Will it be replaced by FX2 media or co-exist with it? -- Fabrizio Giudici - Java Architect @ Tidalwave s.a.s. We make Java work. Everywhere. http://tidalwave.it/fabrizio/**blog http://tidalwave.it/fabrizio/blog - fabrizio.giud...@tidalwave.it
Re: JavaFX Media issues
On 9/08/2013 02:10, Scott Palmer wrote:
The Media APIs are mostly useless in their current state. Other than
demoing that you can play a video, they don't go far enough to be of
practical value. I tried to get someone to pay attention to them back in
the JavaFX 1.0 days
https://javafx-jira.kenai.com/browse/RT-2684
Unfortunately, I have to agree here. I'd love to use the Media APIs of
JavaFX, but they are way to limited. At a minimum I'd need support for
MKV containers and some common audio formats used with these. Decoding
AVI containers would be rather important as well. Without those, JavaFX
video is basically limited to bring your own videos only and
completely unsuitable for playing back videos that end users might have...
Instead, I've been using VLC + VLCJ from day one while working with
JavaFX as a work-around.
at least someone listened to the request to get H.264 support in there, but
that is just a workaround. We need to be able to get our data into the
media pipeline. This would allow those of us that have attempted to do a
video window to have a fighting chance. Canvas can't keep up and will
likely crash the app with out of memory errors. Support for drawing into a
native surface (OpenGL or D3D context) has been talked about, but doesn't
appear to be on the horizon yet. If we just had a hook to get the dang
pixel data into the media pipeline so we could supply the next frame with
whatever we want - either from any native codec via JNI, or dynamically
generated from Java code, whatever... that would be just so dang useful...
(to me at least)
This would atleast allow me to decode the MKV container myself and
supply data... not looking forward to having to write all that code, but
I would if it meant I'd be able to go native JavaFX with video.
Canvas however is working for me even with HD video (copying 25
frames/sec of HD video from VLCJ to Canvas). Playback is smooth even
with 1920x1080 video for hours long video, and I can't tell the
difference with a native player or copying frame by frame using
pixelWriter anymore. There is some CPU penalty but on a my system it is
well below 5%. This is basically how that looks with VLCJ:
new RenderCallback() {
@Override
public void display(DirectMediaPlayer mp, Memory[] memory,
final BufferFormat bufferFormat) {
final ByteBuffer byteBuffer = memory[0].getByteBuffer(0,
memory[0].size());
Platform.runLater(new Runnable() {
@Override
public void run() {
pixelWriter.setPixels(0, 0, bufferFormat.getWidth(),
bufferFormat.getHeight(), byteBgraInstance, byteBuffer,
bufferFormat.getPitches()[0]);
}
});
}
}
However, the bug where Canvas keeps buffering data when off-screen
sometimes bites me, resulting in out of memory errors -- even slight
display delays can fairly easily cause this when video is running at 25
fps (takes about a second to go through 256 MB of memory).
--John
Regards,
Scott
On Thu, Aug 8, 2013 at 5:04 PM, Fabrizio Giudici
fabrizio.giud...@tidalwave.it wrote:
On Thu, 08 Aug 2013 22:57:51 +0200, Joe McGlynnjoe.mcgl...@oracle.com
wrote:
I don't know why FX Media isn't in the FX 8 API docs, but that's clearly
an error. Please file a bug on that.
In the meantime, you should look at the FX 2 media docs, there isn't a
lot of change from FX2 media in FX8. Buffering and streaming (HTTP Live
Streaming) are both supported, as is playback from a URL.
What is the strategy for codecs? I mean, now we have ImageIO (there is
also JAI but it seems basically dead). ImageIO provides many image codecs
and there's a SPI that can be used to support more formats. Will it be
replaced by FX2 media or co-exist with it?
--
Fabrizio Giudici - Java Architect @ Tidalwave s.a.s.
We make Java work. Everywhere.
http://tidalwave.it/fabrizio/**bloghttp://tidalwave.it/fabrizio/blog -
fabrizio.giud...@tidalwave.it
Re: JavaFX Media issues
At minimum the Media Player must support H.264, MP3, AAC (so there is a
guarantee of a cross-platform format) and then defer to the default
platform media decoder for everything else. I.e. *anything* that the
standard OS provided media system can decode - I.e. QuickTime on Mac,
GStreamer on Linux, DirectShow on Windows. But all that is needed is the
hook and the community can make the bindings to QuickTime, VLC, etc.
The current implementation appears to be too unstable to rely on at the
moment anyway though: https://javafx-jira.kenai.com/browse/RT-31412
I have heard rumors of people being able to play HD video via Canvas. I
have tried everything and can't come close. (Yes, I have been careful about
the pixel format.) I mean, it looks like it is working for a few seconds,
but then as the memory fills with the Canvas backlog it can lead to the GC
using a lot more CPU, thus reducing the ability for Canvas to process its
command queue even further, well it just collapses in on itself and dies.
Is your app able to do *anything* else while the video is playing? The
slightest delay to the rendering and that Canvas buffering bug kills the
app. Not that it would matter if it could keep up, because the off-screen
thing is also a deal breaker.
Of course 25fps is well below the 60fps required for full-speed video. I
suspect it is the frame rate more than the frame size that matters here.
Plain old, standard definition, interlaced, 60 fields per second will
probably kill most apps with the current Canvas implementation.
We need something better. I proposed having at least a JNI method so we
could get native window handles from Stages but didn't get any traction
either. Security was brought up as a concern, which I totally do NOT
understand as the use of JNI means you are out of the Java sandbox already.
If we had native window handles we might be able get our own window for
rendering to at least interact nicely with the Stage. (We already did this
successfully in Swing via JAWT and a heavyweight component that we paint to
from native code)
Regards,
Scott
On Thu, Aug 8, 2013 at 9:06 PM, John Hendrikx hj...@xs4all.nl wrote:
On 9/08/2013 02:10, Scott Palmer wrote:
The Media APIs are mostly useless in their current state. Other than
demoing that you can play a video, they don't go far enough to be of
practical value. I tried to get someone to pay attention to them back in
the JavaFX 1.0 days
https://javafx-jira.kenai.com/**browse/RT-2684https://javafx-jira.kenai.com/browse/RT-2684
Unfortunately, I have to agree here. I'd love to use the Media APIs of
JavaFX, but they are way to limited. At a minimum I'd need support for MKV
containers and some common audio formats used with these. Decoding AVI
containers would be rather important as well. Without those, JavaFX video
is basically limited to bring your own videos only and completely
unsuitable for playing back videos that end users might have...
Instead, I've been using VLC + VLCJ from day one while working with JavaFX
as a work-around.
at least someone listened to the request to get H.264 support in there,
but
that is just a workaround. We need to be able to get our data into the
media pipeline. This would allow those of us that have attempted to do a
video window to have a fighting chance. Canvas can't keep up and will
likely crash the app with out of memory errors. Support for drawing into
a
native surface (OpenGL or D3D context) has been talked about, but doesn't
appear to be on the horizon yet. If we just had a hook to get the dang
pixel data into the media pipeline so we could supply the next frame
with
whatever we want - either from any native codec via JNI, or dynamically
generated from Java code, whatever... that would be just so dang useful...
(to me at least)
This would atleast allow me to decode the MKV container myself and supply
data... not looking forward to having to write all that code, but I would
if it meant I'd be able to go native JavaFX with video.
Canvas however is working for me even with HD video (copying 25 frames/sec
of HD video from VLCJ to Canvas). Playback is smooth even with 1920x1080
video for hours long video, and I can't tell the difference with a native
player or copying frame by frame using pixelWriter anymore. There is some
CPU penalty but on a my system it is well below 5%. This is basically how
that looks with VLCJ:
new RenderCallback() {
@Override
public void display(DirectMediaPlayer mp, Memory[] memory, final
BufferFormat bufferFormat) {
final ByteBuffer byteBuffer = memory[0].getByteBuffer(0,
memory[0].size());
Platform.runLater(new Runnable() {
@Override
public void run() {
pixelWriter.setPixels(0, 0, bufferFormat.getWidth(),
bufferFormat.getHeight(), byteBgraInstance, byteBuffer,
bufferFormat.getPitches()[0]);
}
