Re: Re: Re: Implementing OpenType font support, how hard?
On 8/4/06, Bertrand Delacretaz [EMAIL PROTECTED] wrote: ...I'll discuss the plan with my project's stakeholders and hopefully get the green light to invest time in this... Good news, I got the green light and will be working on these OpenType improvements in the next few weeks. I'll use the dependencies of issue 40464 [1] to document my progress, and of course discuss the details here. -Bertrand [1] http://issues.apache.org/bugzilla/showdependencytree.cgi?id=40464
Re: Re: Implementing OpenType font support, how hard?
On 8/3/06, Jeremias Maerki [EMAIL PROTECTED] wrote: ...Sorry, I haven't been clear. Adam obviously grabbed a class [1] from Victor's FOray, adapted it to FOP and put a different license header on top. So, it's not that simple. As a first point, we'll need a license grant from Victor for this file or get him to commit it himself to FOP under his ICLA or we use his file and modify it without removing the original license header (I don't like that last option). Not sure how best to deal with this Ok, thanks for the clarification. I'll see if Victor would agree to donate this one class as a temporary solution, until FOP uses FOray. If not I'll write another implementation. Again, thanks everybody for your comments - I'll discuss the plan with my project's stakeholders and hopefully get the green light to invest time in this. -Bertrand
Re: Implementing OpenType font support, how hard?
On Wed, Aug 02, 2006 at 06:44:28PM +0200, Vincent Hennebert wrote: 3) Additional steps for OpenType GSUB table support The goal is to enable the smart font features of OpenType, automatic ligatures as mentioned above, language-dependent glyph substitutions (different shapes if a letter is at the beginning of a word for example), automatic decorative swashes at the beginning or end of words, etc. 3b) Modify the chars-to-metrics mapping to handle things like automatic ligatures, where several chars map to a single glyph Here I think you can profit from my work on kerning to handle special cases. The only problem I see with ligatures is when a word may be hyphenated between two characters for which there is a ligature: if it ends up being hyphenated the separate glyphs should be used, otherwise the ligature glyph should be used. I don't think this can be easily represented in the current Knuth glue/box/penalty model which is used to break lines into paragraphs. I think it can be done, and is similar to things that have been done in FOP. The main problem with all these smart font features is that you cannot implement them in rendering without also implementing them in the linebreaking code. The linebreaking code must be able to correctly predict the metrics of the characters that are going to be rendered. 3d) Create test documents to demonstrate this, asking a font provider for a donation of some OpenType fonts to use in FOP tests. That's one possibility. Another one might be the DejaVu fonts which we have found after a LOT of searching for a font with an ASF-compatible license. However, I haven't received any official feedback on license compatibility, yet. OTOH, I'm not sure if those fonts will enable you to show off all the features you want to implement. Aren't DejaVu fonts only TrueType fonts? Have a look at the STIX fonts, http://www.stixfonts.org/swdev_geninfo.html. I believe they will distribute OpenType fonts with Type1 outlines. Their pace of development is quite slow, but now they are nearing completion of the project. Regards, Simon -- Simon Pepping home page: http://www.leverkruid.eu
Re: Implementing OpenType font support, how hard?
On Thursday 03 August 2006 21:04, Simon Pepping wrote: On Wed, Aug 02, 2006 at 06:44:28PM +0200, Vincent Hennebert wrote: snip/ 3b) Modify the chars-to-metrics mapping to handle things like automatic ligatures, where several chars map to a single glyph Here I think you can profit from my work on kerning to handle special cases. The only problem I see with ligatures is when a word may be hyphenated between two characters for which there is a ligature: if it ends up being hyphenated the separate glyphs should be used, otherwise the ligature glyph should be used. I don't think this can be easily represented in the current Knuth glue/box/penalty model which is used to break lines into paragraphs. I think it can be done, and is similar to things that have been done in FOP. The main problem with all these smart font features is that you cannot implement them in rendering without also implementing them in the linebreaking code. The linebreaking code must be able to correctly predict the metrics of the characters that are going to be rendered. That comment does not only apply to line breaking, justification, hyphenation, word spacing, are all affected. That is layout needs to know the exact metrics the renderer is going to use. snip/ Regards, Simon Manuel
Re: Implementing OpenType font support, how hard?
On 03.08.2006 15:04:40 Simon Pepping wrote: snip/ Aren't DejaVu fonts only TrueType fonts? I think so, but they still have tables for some of the features Bertrand is investigating to implement. Have a look at the STIX fonts, http://www.stixfonts.org/swdev_geninfo.html. I believe they will distribute OpenType fonts with Type1 outlines. Their pace of development is quite slow, but now they are nearing completion of the project. Hmm, not much meat on that bone. The license at least looks promising and compatible with ASF policy (pending confirmation from the ASF's legal VP). Jeremias Maerki
Re: Re: Implementing OpenType font support, how hard?
Hi, and thanks everybody for your replies. On 8/2/06, Jeremias Maerki [EMAIL PROTECTED] wrote: ...No, I've been able to restore kerning support. If there's still some commented code I should probably remove it now. Can you give me a pointer?... You're right, kerning works for builtin fonts at least. It doesn't seem to work in my tests with user-specified fonts, but I've just done a quick test. ...I wonder if supporting Type 1 outlines would be worth the effort. So far, I've never seen an OpenType font with Type 1 outlines. Have you?... Seems like most of the standard OpenType fonts on my MacOSX system use Type 1 outlines. But you're right that supporting TrueType outlines would be a good start already. 2c) Verify that the character encodings are correct... ...The problem is that FOP does not currently support generating a ToUnicode table. Victor Mote has the fixed in FOray and we have a patch in Bugzilla that uses that code to do the same for FOP. Since nobody has dealt with the legal part of grabbing someone else's code in this case, the patch hasn't been applied, yet. The patch: http://issues.apache.org/bugzilla/show_bug.cgi?id=5335 IIUC what's needed here is to contact Adam Strzelecki, author of the patch, and ask him to Grant license to ASF for inclusion in ASF works (as per the Apache Software License §5)? This is how things are done when patches are uploaded via http://issues.apache.org/jira. ...Finishing 2) would then also mean finishing FOrayFont to the degree that it can be used in FOP. I guess that will need further deliberation... I've studied FOray and integrating it is probably out of scope for my current work. It doesn't look too hard, but it impacts quite a lot of existing code, so I fear there might be hidden roadbumps in there. OTOH I agree that using it (and maybe re-integrating that code in FOP?) seems to make more sense than doing a lot of work on FOP's current font-handling code. Also, from other's comments in this thread, it seems like handling smart font features (glyph substitutions) might be a lot of work, better done in a (yet hypothetical) second phase once basic OpenType support works well. At this point, my plan, for a first phase, would be -Integrate Adam Strzelecki's patch to support extended character sets cleanly -Check that OpenType fonts with TrueType outlines are usable as custom fonts, including kerning. Fix things as needed, and have a look at what's needed to support Type 1/CFF outlines. -Try to get a few OpenType reference fonts that can be distributed with FOP for testing and demonstration (I'd ask some font editors, the fonts can be crippled if they want, for example by removing portions of the glyph sets). The smart fonts stuff would (maybe) come later, the above might be sufficient for my current project. I'm being much less ambitious than yesterday...but the above looks like a useful concrete step. WDYT? -Bertrand
Re: Re: Implementing OpenType font support, how hard?
On 8/3/06, Manuel Mall [EMAIL PROTECTED] wrote: On Thursday 03 August 2006 21:04, Simon Pepping wrote: ... The main problem with all these smart font features is that you cannot implement them in rendering without also implementing them in the linebreaking code ...That comment does not only apply to line breaking, justification, hyphenation, word spacing, are all affected. That is layout needs to know the exact metrics the renderer is going to use. Doesn't kerning cause the same problem as smart font features (like automatic ligatures)? It also causes the total width of a group of character to change when the group is split between two lines. -Bertrand
Re: Implementing OpenType font support, how hard?
On 03.08.2006 17:40:36 Bertrand Delacretaz wrote: Hi, and thanks everybody for your replies. On 8/2/06, Jeremias Maerki [EMAIL PROTECTED] wrote: ...No, I've been able to restore kerning support. If there's still some commented code I should probably remove it now. Can you give me a pointer?... You're right, kerning works for builtin fonts at least. It doesn't seem to work in my tests with user-specified fonts, but I've just done a quick test. It worked for me last time I checked, even with user-specified fonts if kerning=yes in the font config and if the XML font metric file contains the kerning info. ...I wonder if supporting Type 1 outlines would be worth the effort. So far, I've never seen an OpenType font with Type 1 outlines. Have you?... Seems like most of the standard OpenType fonts on my MacOSX system use Type 1 outlines. But you're right that supporting TrueType outlines would be a good start already. 2c) Verify that the character encodings are correct... ...The problem is that FOP does not currently support generating a ToUnicode table. Victor Mote has the fixed in FOray and we have a patch in Bugzilla that uses that code to do the same for FOP. Since nobody has dealt with the legal part of grabbing someone else's code in this case, the patch hasn't been applied, yet. The patch: http://issues.apache.org/bugzilla/show_bug.cgi?id=5335 IIUC what's needed here is to contact Adam Strzelecki, author of the patch, and ask him to Grant license to ASF for inclusion in ASF works (as per the Apache Software License §5)? This is how things are done when patches are uploaded via http://issues.apache.org/jira. Sorry, I haven't been clear. Adam obviously grabbed a class [1] from Victor's FOray, adapted it to FOP and put a different license header on top. So, it's not that simple. As a first point, we'll need a license grant from Victor for this file or get him to commit it himself to FOP under his ICLA or we use his file and modify it without removing the original license header (I don't like that last option). Not sure how best to deal with this. [1] http://svn.sourceforge.net/viewvc/foray/trunk/foray/foray-pdf/src/java/org/foray/pdf/object/PDFToUnicodeCMap.java?revision=6314view=markup ...Finishing 2) would then also mean finishing FOrayFont to the degree that it can be used in FOP. I guess that will need further deliberation... I've studied FOray and integrating it is probably out of scope for my current work. It doesn't look too hard, but it impacts quite a lot of existing code, so I fear there might be hidden roadbumps in there. OTOH I agree that using it (and maybe re-integrating that code in FOP?) seems to make more sense than doing a lot of work on FOP's current font-handling code. Also, from other's comments in this thread, it seems like handling smart font features (glyph substitutions) might be a lot of work, better done in a (yet hypothetical) second phase once basic OpenType support works well. At this point, my plan, for a first phase, would be -Integrate Adam Strzelecki's patch to support extended character sets cleanly -Check that OpenType fonts with TrueType outlines are usable as custom fonts, including kerning. Fix things as needed, and have a look at what's needed to support Type 1/CFF outlines. -Try to get a few OpenType reference fonts that can be distributed with FOP for testing and demonstration (I'd ask some font editors, the fonts can be crippled if they want, for example by removing portions of the glyph sets). +1 provided we deal with the ToUnicode stuff properly. The smart fonts stuff would (maybe) come later, the above might be sufficient for my current project. I'm being much less ambitious than yesterday...but the above looks like a useful concrete step. WDYT? -Bertrand Jeremias Maerki
Re: Re: Implementing OpenType font support, how hard?
On Thu, Aug 03, 2006 at 05:40:36PM +0200, Bertrand Delacretaz wrote: At this point, my plan, for a first phase, would be -Integrate Adam Strzelecki's patch to support extended character sets cleanly -Check that OpenType fonts with TrueType outlines are usable as custom fonts, including kerning. Fix things as needed, and have a look at what's needed to support Type 1/CFF outlines. -Try to get a few OpenType reference fonts that can be distributed with FOP for testing and demonstration (I'd ask some font editors, the fonts can be crippled if they want, for example by removing portions of the glyph sets). The smart fonts stuff would (maybe) come later, the above might be sufficient for my current project. I'm being much less ambitious than yesterday...but the above looks like a useful concrete step. WDYT? This sounds very useful to me. Regards, Simon -- Simon Pepping home page: http://www.leverkruid.eu
Re: Implementing OpenType font support, how hard?
Bertrand Delacretaz wrote: Doesn't kerning cause the same problem as smart font features (like automatic ligatures)? It also causes the total width of a group of character to change when the group is split between two lines. AFAICT kerning rarely applies to character pairs which may be subject to line breaks, at least for western alphabets. So I wouldn't worry all that much, except for proper adjustments in case of hyphenation. Arabic, thai and various indian scripts are another matter, but then there are a lot of more basic features for handling these scripts properly missing anyway. J.Pietschmann
Re: Implementing OpenType font support, how hard?
Salut Bertrand, good to see you on this list again. On 02.08.2006 16:33:59 Bertrand Delacretaz wrote: Hi FOP team, (sorry about the long message, but there's quite a lot of stuff to explain) I *might* have a need and budget to improve the support of OpenType fonts in FOP in the next few weeks. We had a quick talk about this with Jeremias at ApacheCon EU, and now I need to estimate how hard/long this could be. I'll list the various steps that I imagine are needed. Please bear with me if there's some sillyness below, I don't know much about how FOP handles fonts so I might be totally off on some assumptions. My target output is only PDF at this point, I haven't looked at the issues for other formats. If you're not familiar with what OpenType brings to fonts (in addition to extended character sets, and usually high quality fonts), the BelloPro tester [2] is quite convincing IMHO. Feedback and alternative ideas are welcome! 1) My understanding of the current situation. FOP allows user-specified fonts to be used and embedded in the generated PDF. Provided utilities (font file readers) convert PostScript Type 1 and TrueType fonts to XML files storing font metrics and kerning information. The TrueType utility should be able to handle OpenType fonts which contain TrueType outlines (in my tests it worked with some fonts, but PDF embedding was not correct for the MacOSX Preview app, although Adobe Reader was fine). OpenType fonts with Type 1 outlines are not usable in FOP currently. At the layout stage, FOP uses static font metrics to find the size of a character: FOP asks the Font object what are the metrics of this character. This works one character at a time, which would prevent a smart Font from performing ligature substitutions, for example (where e.g. ffi is mapped to a single glyph). When generating PDF, FOP embeds the font, subsetting it if it's a TrueType font (e.g. storing only glyphs that are actually used in the document). So far, that's correct. Kerning does not work in the FOP trunk currently, the corresponding code is commented out. No, I've been able to restore kerning support. If there's still some commented code I should probably remove it now. Can you give me a pointer? 2) Steps for simple support of OpenType fonts 2a) Write an OpenTypeFontFileReader, factoring out and reusing code of the TrueType and Type 1 readers to support OpenType fonts having either Type 1 or TrueType outlines. I wonder if supporting Type 1 outlines would be worth the effort. So far, I've never seen an OpenType font with Type 1 outlines. Have you? 2b) Complete the font embedding code so that it works for both OpenType outline variants (maybe ignoring subsetting if it makes it easier) Subsetting is not that complicated I think. 2c) Verify that the character encodings are correct when embedding the font, might need improvements? [1] seems to imply that this is problematic currently. The problem is that FOP does not currently support generating a ToUnicode table. Victor Mote has the fixed in FOray and we have a patch in Bugzilla that uses that code to do the same for FOP. Since nobody has dealt with the legal part of grabbing someone else's code in this case, the patch hasn't been applied, yet. The patch: http://issues.apache.org/bugzilla/show_bug.cgi?id=5335 2d) Re-enable kerning, as OpenType fonts are usually of high quality and deserve to be used with automatic kerning. Ok, that should be obsolete. One point about 2) is that Vincent Hennebert and Victor Mote are working on FOrayFont to create a better font library which we'd like to use when it's finished. So this may mean that some of this work would better be done in/for ForayFont. Finishing 2) would then also mean finishing FOrayFont to the degree that it can be used in FOP. I guess that will need further deliberation. 3) Additional steps for OpenType GSUB table support The goal is to enable the smart font features of OpenType, automatic ligatures as mentioned above, language-dependent glyph substitutions (different shapes if a letter is at the beginning of a word for example), automatic decorative swashes at the beginning or end of words, etc. 3a) Decode the GSUB table of the OpenType font (and other tables that might be required to use it) and store its data in the FOP XML font metrics file One goal of FOrayFont is to make the separate metrics file obsolete. The font files will be interpreted directly. This should also simplify the whole system, especially for the user. 3b) Modify the chars-to-metrics mapping to handle things like automatic ligatures, where several chars map to a single glyph Here I think you can profit from my work on kerning to handle special cases. 3c) Implement GSUB table handling, glyph substitutions (or reuse an existing library for this, but the only one that I've found is freetype, haven't found one in Java). 3d) Create test
Re: Re: Implementing OpenType font support, how hard?
Hoi Jeremias, I'll reply on the other points tomorrow, but for now: ...AFAIK, OpenType allows different variants of a font in one font file (ex. normal and bold). We've had requests to support those font files. Have you found out during your investigations what would be involved in supporting this and would this be in scope for your work?... AFAIK, several OpenType fonts can be packaged in a single file as TrueType Collections with a .TTC file extension. If that's what you're thinking about, extracting the individual fonts from such a file shouldn't be a problem. -Bertrand
Re: Implementing OpenType font support, how hard?
Hi Bertrand, As I've made some work in this area, I can provide a few additional hints. In fact I'm kind of a bridge between FOray and Fop and am working on adapting FOrayFont to Fop. Currently I'm not doing much because I'm busy with some other work on Fop for the Google Summer of Code, but I should have some time again to work on this from mid-september on (mmmh, is it too late for you?). snip/ 2d) Re-enable kerning, as OpenType fonts are usually of high quality and deserve to be used with automatic kerning. Ok, that should be obsolete. One point about 2) is that Vincent Hennebert and Victor Mote are working on FOrayFont to create a better font library which we'd like to use when it's finished. So this may mean that some of this work would better be done in/for ForayFont. Finishing 2) would then also mean finishing FOrayFont to the degree that it can be used in FOP. I guess that will need further deliberation. (Some quick background on this: I submitted a patch in december 2005 which integrated FOrayFont into Fop; it was not applied because of too severe limitations of FOrayFont. I'm currently working on implementing the missing features; there are still two of them to implement, and then the adaptation may be restarted. Of course the Fop code has quite evolved since december.) I believe there is basic OpenType support in FOrayFont, I can't say more without having a deeper look. AFAICT there are two areas of work: * complete the parsing of OpenType font files; * make sure the API provides access to the most advanced features of OpenType fonts. The FOray project: http://foray.sourceforge.net/ You may be interested in subscribing to the foray-dev list. 3) Additional steps for OpenType GSUB table support The goal is to enable the smart font features of OpenType, automatic ligatures as mentioned above, language-dependent glyph substitutions (different shapes if a letter is at the beginning of a word for example), automatic decorative swashes at the beginning or end of words, etc. 3a) Decode the GSUB table of the OpenType font (and other tables that might be required to use it) and store its data in the FOP XML font metrics file One goal of FOrayFont is to make the separate metrics file obsolete. The font files will be interpreted directly. This should also simplify the whole system, especially for the user. 3b) Modify the chars-to-metrics mapping to handle things like automatic ligatures, where several chars map to a single glyph Here I think you can profit from my work on kerning to handle special cases. The only problem I see with ligatures is when a word may be hyphenated between two characters for which there is a ligature: if it ends up being hyphenated the separate glyphs should be used, otherwise the ligature glyph should be used. I don't think this can be easily represented in the current Knuth glue/box/penalty model which is used to break lines into paragraphs. I also believe that in German (and perhaps other languages), ligatures are not welcome in some compound words. And I wonder if the ligature mechanism is similar for non-western languages which heavily use them (Arabic, for example). 3c) Implement GSUB table handling, glyph substitutions (or reuse an existing library for this, but the only one that I've found is freetype, haven't found one in Java). 3d) Create test documents to demonstrate this, asking a font provider for a donation of some OpenType fonts to use in FOP tests. That's one possibility. Another one might be the DejaVu fonts which we have found after a LOT of searching for a font with an ASF-compatible license. However, I haven't received any official feedback on license compatibility, yet. OTOH, I'm not sure if those fonts will enable you to show off all the features you want to implement. Aren't DejaVu fonts only TrueType fonts? Even this wouldn't be complete, as OpenType allows specific features to be enabled for specific character runs, like use alternate glyph set 2 for this character only. But it would be a good start already ;-) :-) Sure. At this point I'm mostly interested in your opinion on points 1) and 2) above, if these enhancements seem realistic I might be able to work on them in my current project. Point 3) obviously needs more work and might not fit my budget at this point. In general, I'm happy if we get some reinforcements on the font front. 2) shouldn't be a very big task. But I assume the whole FOrayFont thing might make this a little more complicated. AFAIK, OpenType allows different variants of a font in one font file (ex. normal and bold). We've had requests to support those font files. Have you found out during your investigations what would be involved in supporting this and would this be in scope for your work? So far, I've been unable to find out how this is handled. Thanks for any feedback on this! -Bertrand [1] http://xmlgraphics.apache.org/fop/0.20.5/fonts.html#embedding [2]
