Re: [Mlt-devel] Xml output is currenty broken
On Wed, Jan 25, 2012 at 9:42 PM, Brian Matherly pez4br...@yahoo.com wrote: I can provide a patch to convert the entire module to use mlt_log() later. And here it is: https://github.com/pez4brian/mlt/commit/dc854a78f229b7218990cb742a8a507fba457842 applied -- +-DRD-+ -- Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 ___ Mlt-devel mailing list Mlt-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mlt-devel
Re: [Mlt-devel] Xml output is currenty broken
On Wed, Jan 25, 2012 at 8:21 PM, Brian Matherly pez4br...@yahoo.com wrote: Dan, I made my comments on github. Thanks Dan. Let me rework the patch and get back to you. I think I will: * Put the startElement callback back where it belongs. * Add code to bail out after the first pass if there is an error. * Add a handler for the warning callback which uses mlt_log_warning(). Have a look at this one: https://github.com/pez4brian/mlt/commit/fb55f5e0af62c4d9a713eea34587bd91e04601c6 It handles errors, warnings and fatals. It also skips the second pass if the first pass fails. I can provide a patch to convert the entire module to use mlt_log() later. ~BM That looks great. I merged it. -- +-DRD-+ -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Mlt-devel mailing list Mlt-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mlt-devel
Re: [Mlt-devel] Xml output is currenty broken
On Mon, Jan 23, 2012 at 7:58 PM, Brian Matherly pez4br...@yahoo.com wrote: Dan, I did come up with some additional error reporting so that if the xml fails to load, at least it doesn't fail silently: https://github.com/pez4brian/mlt/commit/5fc4d19e81e658e1236da5b82457cfa8b428a705 Feel free to pull it if you like. For JB's example file, it prints an error like this: XML parse error: PCDATA invalid Char value 11 row: 21 col: 20 XML parse error: PCDATA invalid Char value 11 row: 30 col: 20 I definitely like the contribution, but it should use mlt_log_warning() instead of fprintf(stderr). The other fprintf(stderr) you see there are simply because that file was not yet updated to use mlt_log. My policy is that any new code or code changed around a legacy approach should adopt the new approach instead of deferring to consistency with the older code in a file. (Same goes for other things like no longer comparing pointers with NULL.) However, a single commit should not include both a logic change and a comprehensive update to the new approach. Here is the new error message. https://github.com/pez4brian/mlt/commit/04df8a240c38d8efed5a822f01f8bb074ace5a01 I made my comments on github. -- +-DRD-+ -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Mlt-devel mailing list Mlt-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mlt-devel
Re: [Mlt-devel] Xml output is currenty broken
I did come up with some additional error reporting so that if the xml fails to load, at least it doesn't fail silently: snip I definitely like the contribution, but it should use mlt_log_warning() instead of fprintf(stderr). The other fprintf(stderr) you see there are simply because that file was not yet updated to use mlt_log. My policy is that any new code or code changed around a legacy approach should adopt the new approach instead of deferring to consistency with the older code in a file. (Same goes for other things like no longer comparing pointers with NULL.) However, a single commit should not include both a logic change and a comprehensive update to the new approach. Here is the new error message. https://github.com/pez4brian/mlt/commit/04df8a240c38d8efed5a822f01f8bb074ace5a01 I made my comments on github. Thanks Dan. Let me rework the patch and get back to you. I think I will: * Put the startElement callback back where it belongs. * Add code to bail out after the first pass if there is an error. * Add a handler for the warning callback which uses mlt_log_warning(). ~Brian -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Mlt-devel mailing list Mlt-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mlt-devel
Re: [Mlt-devel] Xml output is currenty broken
I thought that I could add some sanity checking to producer_xml.c and maybe even fix broken input XML encoding. But libxml doesn't make that simple. I did come up with some additional error reporting so that if the xml fails to load, at least it doesn't fail silently: https://github.com/pez4brian/mlt/commit/5fc4d19e81e658e1236da5b82457cfa8b428a705 Feel free to pull it if you like. For JB's example file, it prints an error like this: XML parse error: PCDATA invalid Char value 11 row: 21 col: 20 XML parse error: PCDATA invalid Char value 11 row: 30 col: 20 I definitely like the contribution, but it should use mlt_log_warning() instead of fprintf(stderr). The other fprintf(stderr) you see there are simply because that file was not yet updated to use mlt_log. My policy is that any new code or code changed around a legacy approach should adopt the new approach instead of deferring to consistency with the older code in a file. (Same goes for other things like no longer comparing pointers with NULL.) However, a single commit should not include both a logic change and a comprehensive update to the new approach. I'll work up a patch tonight to convert the entire xml module to use mlt_log_warning(). ~BM -- Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 ___ Mlt-devel mailing list Mlt-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mlt-devel
Re: [Mlt-devel] Xml output is currenty broken
I thought of a new policy to add to docs/policies.txt and somewhere in doxygen comments: The standard for strings in MLT is UTF-8. Applications must provide valid UTF-8. That means, melt would be responsible for converting from environment locale's encoding to UTF-8. And melt would be responsible for converting from UTF-8 to environment locale's encoding for output. makes sense For dependent libraries, if their API or documentation discloses character the encoding we need to convert it to UTF-8 (and filtered by icon along the way), and if it unknown (e.g. av_dict), then we should assume UTF-8 and filter it. Comments accepted. I changed my mind about this policy because some libc functions like the mbrtowc() I just added uses the locale's encoding. If we require UTF-8 within MLT strings, then any libc function that uses locale would require string conversion from and to UTF-8. Naturally, libc is used much more broadly and pervasively than any other lib. I am now thinking that MLT should be using the current locale's encoding, and if a library like libxml has a specific encoding requirement we should convert it at that interface. I think this policy would reduce the number of overall changes and confine changes to more obvious and well-understood locations. That is a common and valid approach. One small disadvantage is that you can end up with bugs that only affect people in certain locales. In that case, you might receive a bug report and spend some time trying to figure out why the bug does not occur on your system. You may have to ask the user what locale they are running in. For example, some systems can have a filesystem encoding that is different than the default encoding - so a valid file name string may not work properly with fopen(). But even if you set a policy that all input to MLT must be utf-8, you will probably still end up fielding those questions :) I agree, this policy should reduce the number of conversions and should make the required conversions more understandable. The most important thing is that every string within MLT is consistent. We don't want to mix-and-match string encodings internally from one module to the next. ~BM -- Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 ___ Mlt-devel mailing list Mlt-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mlt-devel
Re: [Mlt-devel] Xml output is currenty broken
On Sun, Jan 22, 2012 at 1:36 PM, Brian Matherly pez4br...@yahoo.com wrote: JB, Creating xml files from the avformat producer is currently broken with FFmpeg's recent git. The problem comes from avformat's handler_name metadata, whose value contains invalid characters. The problem only appears when creating xml files, not when outputting to stdout. For example: melt test.mov -consumer xml You might have to provide your test.mov file. I tried to recreate the problem with some of my media files, but none of them have the handler_name metadata. And if I could find one with handler_name, I don't know that it would necessarily have an invalid character. I don't know if Dan has a file lying around that recreates the problem. I guess you could wait for him to check before you upload something. I'm guessing that the best solution might be to fix the xml producer to handle the xml with the invalid character. In that case, we might be able to make some progress if you provide your broken XML file. I have some files that reproduce the problem. I have a flagged email to followup on a bug report related to xml char encoding. Currently, the xml consumer assumes the string data you provide it is already UTF-8 encoded, but some people are providing, for example, KOI8 file names in Russian locales. I can get the environment's locale info to determine that. Now, I need to figure out what the character encoding rules or API are for libav and its version history. -- +-DRD-+ -- Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 ___ Mlt-devel mailing list Mlt-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mlt-devel
Re: [Mlt-devel] Xml output is currenty broken
On Sun, Jan 22, 2012 at 1:48 PM, Dan Dennedy d...@dennedy.org wrote: On Sun, Jan 22, 2012 at 1:36 PM, Brian Matherly pez4br...@yahoo.com wrote: JB, Creating xml files from the avformat producer is currently broken with FFmpeg's recent git. The problem comes from avformat's handler_name metadata, whose value contains invalid characters. The problem only appears when creating xml files, not when outputting to stdout. For example: melt test.mov -consumer xml You might have to provide your test.mov file. I tried to recreate the problem with some of my media files, but none of them have the handler_name metadata. And if I could find one with handler_name, I don't know that it would necessarily have an invalid character. I don't know if Dan has a file lying around that recreates the problem. I guess you could wait for him to check before you upload something. I'm guessing that the best solution might be to fix the xml producer to handle the xml with the invalid character. In that case, we might be able to make some progress if you provide your broken XML file. I have some files that reproduce the problem. I have a flagged email to followup on a bug report related to xml char encoding. Currently, the xml consumer assumes the string data you provide it is already UTF-8 encoded, but some people are providing, for example, KOI8 file names in Russian locales. I can get the environment's locale info to determine that. Now, I need to figure out what the character encoding rules or API are for libav and its version history. OK, av_dict (and its predecessor av_metadata from avformat.h) has neither rules or API for character encoding. Demuxers quite often simply pass up whatever appears in the file. And the offending character in this example is a 0x0b vertical tab. So, I think we need some string filter function. I am looking around for a good practice regarding string filtering. Then, the next question is whether to filter the output of av_dict or filter the input to xmlNewTextChild() and xmlNewProp(). -- +-DRD-+ -- Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 ___ Mlt-devel mailing list Mlt-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/mlt-devel
