Re: [ft-devel] WOFF2 Support Update
> Is this only useful for WOFF2? Or, can it be productively utilized by > TTC in general? Sorry I missed this mail. Right now this is only useful for WOFF2. In the future, if we have another sfnt-wrapper format that supports TTCs, this can be generalized. -- Nikhil ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 Support Update
On Mon, Aug 5, 2019 at 3:31 AM Werner LEMBERG wrote: > > > > Instead of handling all of this in `woff2_open_font', I have > > introduced a new variable in `sfnt_open_font' (woff2_num_faces), and > > I use this to set `face->root.num_faces' if the input font is a > > WOFF2. (I believe) this solves all the issues. I've implemented > > this in commit b25c352. Please take a look. > > Looks good, thanks! Is this only useful for WOFF2? Or, can it be productively utilized by TTC in general? ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 Support Update
> Instead of handling all of this in `woff2_open_font', I have > introduced a new variable in `sfnt_open_font' (woff2_num_faces), and > I use this to set `face->root.num_faces' if the input font is a > WOFF2. (I believe) this solves all the issues. I've implemented > this in commit b25c352. Please take a look. Looks good, thanks! Werner ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 Support Update
> > So are you suggesting that `sfwoff2.c' directly output the total > > number of faces and named instances (if the index is < 0) and exit > > with error so that FT_Open_Face does not try to read and output this > > data again? > > In case this makes life simpler, please do it! The alternative as > described by you seems to be much slower, which probably makes it > worth to circumvent it. And maybe you can do some code sharing by > shuffling stuff around. However, shuffling is not that important in > case it would be too cumbersome. Instead of handling all of this in `woff2_open_font', I have introduced a new variable in `sfnt_open_font' (woff2_num_faces), and I use this to set `face->root.num_faces' if the input font is a WOFF2. (I believe) this solves all the issues. I've implemented this in commit b25c352. Please take a look. -- Nikhil ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 Support Update
> So are you suggesting that `sfwoff2.c' directly output the total > number of faces and named instances (if the index is < 0) and exit > with error so that FT_Open_Face does not try to read and output this > data again? In case this makes life simpler, please do it! The alternative as described by you seems to be much slower, which probably makes it worth to circumvent it. And maybe you can do some code sharing by shuffling stuff around. However, shuffling is not that important in case it would be too cumbersome. Werner ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 Support Update
> What am I missing? My main concern is that *after* `woff2_open_font' returns, `sfnt_init_face' attempts to read the TTC header and extract information about the total number of faces, and stores this in `face->num_faces', which will override the value stored by `woff2_open_font'. -- Nikhil ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 Support Update
> > This means that you can quickly extract the number of TTC subfonts, > right? Yes, this value is also stored in `WOFF2_HeaderRec' as `num_fonts'. > > > One solution is to load all the faces in the WOFF2 and write all > > tables along with the TTC header. This may slow down the process if > > all the faces have a transformed glyf or hmtx table. > > It's not clear to me why you have to load all faces. As far as I can > see, you need two calls. The first call is to load the TTC header to > get the number of subfonts. This value could be stored somewhere to > cache it for the `index = -1' case. > > Another call (for index < -1 case) then investigates the subfont in > question to find out the number of named instances. > > What am I missing? Yes, this is exactly what I'm doing right now (locally, not pushed yet), except for finding the number of named instances in the WOFF2 face. So are you suggesting that `sfwoff2.c' directly output the total number of faces and named instances (if the index is < 0) and exit with error so that FT_Open_Face does not try to read and output this data again? -- Nikhil ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 Support Update
> `woff2_open_font' reads the WOFF2 TTC header and uses that > information to get tables for the requested font index. This means that you can quickly extract the number of TTC subfonts, right? >> However, to get the number of named instances for a variation font >> with index N, loading this font should be sufficient. You still >> have to get the number of faces in a TTC just to find out whether >> the requested subfont can be loaded at all. > > The issue, however, is that when the font is loaded (when > `woff2_open_font' returns), the `face' will contain the header and > tables for a single face, which means the value of > `face_instance_index' is changed to 0 in every case. > > This means that after this, there is no information left about `n' > where n is the requested index number. > > For example, if I call `FT_New_Face' with -2 (which should ideally > output the following) for a TTC, I get: > > FT_Open_Face: Creating glyph slot > FT_Open_Face: The font has 3 faces > and 0 named instances for face 1 > FT_Open_Face: Return 0x0 > > But doing the same with the font converted to WOFF2, I will get: > > FT_Open_Face: The font has 1 face > and 0 named instances for face 1 > FT_Open_Face: Return 0x0 > > Although the information about the named instances is correct, > `FT_Open_Face' will see only 1 face in the font and report it as > such. > > One solution is to load all the faces in the WOFF2 and write all > tables along with the TTC header. This may slow down the process if > all the faces have a transformed glyf or hmtx table. It's not clear to me why you have to load all faces. As far as I can see, you need two calls. The first call is to load the TTC header to get the number of subfonts. This value could be stored somewhere to cache it for the `index = -1' case. Another call (for index < -1 case) then investigates the subfont in question to find out the number of named instances. What am I missing? Werner ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 Support Update
> Hmm. The functionality of negative `face_index' values as document > in the description of `FT_Open_Face' must work for WOFF2, too. So, to > get the number of faces in a TTC, you have to read the TTC header. `woff2_open_font' reads the WOFF2 TTC header and uses that information to get tables for the requested font index. > However, to get the number of named instances for a variation font > with index N, loading this font should be sufficient. You still have > to get the number of faces in a TTC just to find out whether the > requested subfont can be loaded at all. The issue, however, is that when the font is loaded (when `woff2_open_font' returns), the `face' will contain the header and tables for a single face, which means the value of `face_instance_index' is changed to 0 in every case. This means that after this, there is no information left about `n' where n is the requested index number. For example, if I call `FT_New_Face' with -2 (which should ideally output the following) for a TTC, I get: FT_Open_Face: Creating glyph slot FT_Open_Face: The font has 3 faces and 0 named instances for face 1 FT_Open_Face: Return 0x0 But doing the same with the font converted to WOFF2, I will get: FT_Open_Face: The font has 1 face and 0 named instances for face 1 FT_Open_Face: Return 0x0 Although the information about the named instances is correct, `FT_Open_Face' will see only 1 face in the font and report it as such. One solution is to load all the faces in the WOFF2 and write all tables along with the TTC header. This may slow down the process if all the faces have a transformed glyf or hmtx table. Is there anything else that can be done? -- Nikhil ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 Support Update
> With commit af7d296 (in branch `GSoC-2019-nikhil'), we can now quite > efficiently convert a WOFF2 to an SFNT font, which can then be > rendered by FreeType. Great! > (1) If the WOFF2 is a TTC, only the requested face is loaded. So, > instead of a TTC, we generate a single-face sfnt with one offset > table, table record and so on... [...] > > The issue with (1) is that if I call `FT_New_Face' for a woff2 file > with a negative index (which is supposed to provide information on > the specific face), the code in `sfobjs.c' will only see a single > face, which means `face->num_faces' will be set to 1. I'm not > really sure how to handle this. Any suggestions? Hmm. The functionality of negative `face_index' values as documented in the description of `FT_Open_Face' must work for WOFF2, too. So, to get the number of faces in a TTC, you have to read the TTC header. However, to get the number of named instances for a variation font with index N, loading this font should be sufficient. You still have to get the number of faces in a TTC just to find out whether the requested subfont can be loaded at all. Werner ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
[ft-devel] WOFF2 Support Update
Hi all, An update on the progress of WOFF2 support: With commit af7d296 (in branch `GSoC-2019-nikhil'), we can now quite efficiently convert a WOFF2 to an SFNT font, which can then be rendered by FreeType. Some points: (1) If the WOFF2 is a TTC, only the requested face is loaded. So, instead of a TTC, we generate a single-face sfnt with one offset table, table record and so on... (2) We're using the `totalSfntSize' (from the specs) as a reference value to allocate memory, and then reallocate if required. This reduces the number of memory reallocations and speeds up the process. The issue with (1) is that if I call `FT_New_Face' for a woff2 file with a negative index (which is supposed to provide information on the specific face), the code in `sfobjs.c' will only see a single face, which means `face->num_faces' will be set to 1. I'm not really sure how to handle this. Any suggestions? -- Nikhil ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 support update and questions
>> `FT_Stream_OpenMemory' is be the right function. After calling >> this, you should use `FT_Stream_EnterFrame' and proceed as usual. > > This is working now. [...] OK, LGTM. Werner ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 support update and questions
> `FT_Stream_OpenMemory' is be the right function. After calling this, > you should use `FT_Stream_EnterFrame' and proceed as usual. This is working now. I think the issue was due to some trivial error I made while creating the stream and passing it to other functions. However, could you please check if what I'm doing now is correct? http://git.savannah.gnu.org/cgit/freetype/freetype2.git/tree/src/sfnt/sfwoff2.c?h=GSoC-2019-nikhil=c325089c2fd005e2c324aa4f3ddb942258aed5f1#n340 (see lines 340-344). I've not used `FT_FRAME_ENTER' as of now (in the loop) because I need to call `FT_READ_XXX' (which is not for frames) for some part of the reconstruction. I'll do that when we're ready to copy the table to the sfnt stream. > > Also, is this the right way of doing it? Is there some other way I > > can read the buffer? > > Is there are problem with `FT_READ_XXX'? Nope. It was definitely an issue with my code :-) Initially, I thought it made sense to have only one `stream' per source file (because most read macros use an implicit stream variable), but having a new stream in a separate function works perfectly. -- Nikhil ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 support update and questions
> I have now added code to uncompress the Brotli-compressed stream. Nice! > Right now, I'm allocating the required memory for the stream, and > storing the output in an FT_Byte array (`uncompressed_buf'). How do > I create an `FT_Stream' from this so that I can read it using the > `FT_READ_XXX' macros? I tried using `FT_Stream_OpenMemory' but that > gives me segmentation faults. `FT_Stream_OpenMemory' is be the right function. After calling this, you should use `FT_Stream_EnterFrame' and proceed as usual. > Also, is this the right way of doing it? Is there some other way I > can read the buffer? Is there are problem with `FT_READ_XXX'? Werner ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 support update and questions
Hi, I have now added code to uncompress the Brotli-compressed stream. Right now, I'm allocating the required memory for the stream, and storing the output in an FT_Byte array (`uncompressed_buf'). How do I create an `FT_Stream' from this so that I can read it using the `FT_READ_XXX' macros? I tried using `FT_Stream_OpenMemory' but that gives me segmentation faults. Also, is this the right way of doing it? Is there some other way I can read the buffer? -- Nikhil ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 support update and questions
> I have added Brotli as a dependency with commit > 8e05ab3e2ebf9d90b6015cb2aa7e789517b293e6. This works on my setup > after installing Brotli and running `sh autogen.sh', `./configure' > in the freetype2 directory. I'm also now able to include > `brotli/decode.h' in `sfwoff2.c'. Looks good, thanks. > Is there anything else that needs to be done for the dependency > (like writing similar checks for CMake)? Since cmake support is optional I suggest to wait with it until you are done with everything else. Additioanlly, Nikolaus is working on improvements of the cmake files right now. Werner ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 support update and questions
Hi, I have added Brotli as a dependency with commit 8e05ab3e2ebf9d90b6015cb2aa7e789517b293e6. This works on my setup after installing Brotli and running `sh autogen.sh', `./configure' in the freetype2 directory. I'm also now able to include `brotli/decode.h' in `sfwoff2.c'. Is there anything else that needs to be done for the dependency (like writing similar checks for CMake)? -- Nikhil ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 support update and questions
>> What do you mean with `access'? `sfnt_open_font' fills up the >> array with all header offsets; it thus doesn't need a face index. > > But the whole process of uncompressing a WOFF2 will take place in > `sfwoff2.c' (woff2_open_font), which is being called from > `sfnt_font_open'. If the function is uncompressing a TTC, it will > need to know the requested index so that the tables for that face > can be picked up and written as an SFNT stream. It seems that you have to adjust `sfnt_font_open' for this particular case to pass the face index as an argument... Werner ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 support update and questions
> What do you mean with `access'? `sfnt_open_font' fills up the array > with all header offsets; it thus doesn't need a face index. But the whole process of uncompressing a WOFF2 will take place in `sfwoff2.c' (woff2_open_font), which is being called from `sfnt_font_open'. If the function is uncompressing a TTC, it will need to know the requested index so that the tables for that face can be picked up and written as an SFNT stream. -- Nikhil ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 support update and questions
> Another question: How do I access the requested `face index'? > `sfnt_open_font' does not have the information as a parameter, but > its caller (`sfnt_init_face' does have the requested index as > `face_instance_index'). What do you mean with `access'? `sfnt_open_font' fills up the array with all header offsets; it thus doesn't need a face index. Werner ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 support update and questions
> A single font from a TTC collection has no references to the TTC as a > whole. So yes, you can completely hide the fact that you are > processing a TTC. Great, so I can load only the requested font. > > Do these `individual' streams function identically to a full `TTC' > > stream? This was further clarification to my question, whether creating a stream with only the requested face would have any difference from creating a TTC stream. That part is clear to me now. Another question: How do I access the requested `face index'? `sfnt_open_font' does not have the information as a parameter, but its caller (`sfnt_init_face' does have the requested index as `face_instance_index'). -- Nikhil ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 support update and questions
> Let's say I want to load the nth font inside a TTC. Would it be > sufficient to write an SFNT stream starting with the Offset Table > (and not the TTC Header) along with only those tables that make up > the nth face? Yes. > To put this in context of WOFF2, I want to know whether I should pass > the SFNT stream of a WOFF2 compressed TTC font with a TTC header and > *all* its tables, or would it be correct/sufficient to write the > stream such that the fact that the WOFF2 is actually a collection, is > hidden from rest of the SFNT processing (they only see the requested > face in the stream). A single font from a TTC collection has no references to the TTC as a whole. So yes, you can completely hide the fact that you are processing a TTC. > Do these `individual' streams function identically to a full `TTC' > stream? This sentence I don't understand. Please elaborate. Werner ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 support update and questions
Hi, > > * For TTC fonts, the `Collection Directory' specifies the index of > > tables that are part of each font in the collection. The > > `face_index' parameter in `FT_New_Face' specifies which face to > > load, so does that mean I can use that to directly extract and > > load only those tables that belong to the font on that index? > > I think so, yes. > > > Do I have to decompress each table and load the whole collection > > as an SFNT? I'm not sure... > > Me neither, sorry. I have never worked with WOFF. Please report any > unclear issues to the WOFF2 standard team so that they can improve the > documentation. Alexei pointed out that all the tables are compressed as a single stream, so it is now clear that the whole stream must be uncompressed. I have another question, unrelated to WOFF2, but about TTCs in general: Let's say I want to load the nth font inside a TTC. Would it be sufficient to write an SFNT stream starting with the Offset Table (and not the TTC Header) along with only those tables that make up the nth face? To put this in context of WOFF2, I want to know whether I should pass the SFNT stream of a WOFF2 compressed TTC font with a TTC header and *all* its tables, or would it be correct/sufficient to write the stream such that the fact that the WOFF2 is actually a collection, is hidden from rest of the SFNT processing (they only see the requested face in the stream). Do these `individual' streams function identically to a full `TTC' stream? -- Nikhil ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 support update and questions
> > * Is the code, in general, acceptable? Am I using the FT_* typedefs > > and macros correctly? > > After a quick inspection, everything looks good, thanks! Don't worry > too much, you will get a better feeling for that stuff soon :-) Great, thanks! It does get better over time :-) -- Nikhil ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 support update and questions
> > Do I have to decompress each > > table and load the whole collection as an SFNT? I'm not sure... > > WOFF1 compresses each table, WOFF2 compresses the entire stream. > https://www.w3.org/TR/WOFF2/#FileStructure > Do you have a choice? Oh, this totally slipped my mind! So after decompressing the stream, do I pick up and load only those tables for that font index? -- Nikhil ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 support update and questions
> Regarding testing, you can try the following: > > 1. otsanitizer has a test data directory with "good" and "bad" fonts: > https://github.com/khaledhosny/ots. Remember to compile your copy of > the library with the `-fsanitize=address,undefined > -fno-sanitize-recover` compiler switches so that you get pretty > explosions if your code does weird things with the input :) > > 2. Use fonttools' WOFF2 compression to compress random fonts yourself: > https://github.com/fonttools/fonttools/blob/master/Snippets/woff2_compress.py I'll try out the test fonts, and check with some more fonts too. Thanks! -- Nikhil ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 support update and questions
> Do I have to decompress each > table and load the whole collection as an SFNT? I'm not sure... WOFF1 compresses each table, WOFF2 compresses the entire stream. https://www.w3.org/TR/WOFF2/#FileStructure Do you have a choice? ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 support update and questions
> I have implemented functionality to read the WOFF2 header, table > directory and collection directory (if the font is a TTC), including > most of the checks (given in the specification) in `sfwoff2.c' on > the `GSoC-2019-nikhil' branch. Nice! > It would be *great* if you could review (and also test with a few > font files :-) ) the code and provide feedback! Will do that today or tomorrow, hopefully. > * The code in function `woff2_open_font' is already (almost) 300 > lines. I believe it would be better to split it up into smaller > functions (for instance, one for reading the table directory). > Any thoughts? This is up to you. If you can split the stuff into logical chunks that are easier to understand, please do so. However, this is not a necessity. > * For TTC fonts, the `Collection Directory' specifies the index of > tables that are part of each font in the collection. The > `face_index' parameter in `FT_New_Face' specifies which face to > load, so does that mean I can use that to directly extract and > load only those tables that belong to the font on that index? I think so, yes. > Do I have to decompress each table and load the whole collection > as an SFNT? I'm not sure... Me neither, sorry. I have never worked with WOFF. Please report any unclear issues to the WOFF2 standard team so that they can improve the documentation. > * I have written a few macros in `sfwoff2.c' to help with basic > calculation and to allow implicit variables. Should I `#undef' > these macros towards the end? Well, I guess `sfwoff2.c' gets included with other .c files in `sfnt.c', so it probably makes sense to undefine them, yes. > * Is the code, in general, acceptable? Am I using the FT_* typedefs > and macros correctly? After a quick inspection, everything looks good, thanks! Don't worry too much, you will get a better feeling for that stuff soon :-) Werner ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
Re: [ft-devel] WOFF2 support update and questions
Regarding testing, you can try the following: 1. otsanitizer has a test data directory with "good" and "bad" fonts: https://github.com/khaledhosny/ots. Remember to compile your copy of the library with the `-fsanitize=address,undefined -fno-sanitize-recover` compiler switches so that you get pretty explosions if your code does weird things with the input :) 2. Use fonttools' WOFF2 compression to compress random fonts yourself: https://github.com/fonttools/fonttools/blob/master/Snippets/woff2_compress.py ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
[ft-devel] WOFF2 support update and questions
Hi folks, I have implemented functionality to read the WOFF2 header, table directory and collection directory (if the font is a TTC), including most of the checks (given in the specification) in `sfwoff2.c' on the `GSoC-2019-nikhil' branch. It would be *great* if you could review (and also test with a few font files :-) ) the code and provide feedback! The trace output is quite verbose at level 2 and returns `Unimplemented_Feature' after execution. I have a few observations and questions: * The code in function `woff2_open_font' is already (almost) 300 lines. I believe it would be better to split it up into smaller functions (for instance, one for reading the table directory). Any thoughts? * For TTC fonts, the `Collection Directory' specifies the index of tables that are part of each font in the collection. The `face_index' parameter in `FT_New_Face' specifies which face to load, so does that mean I can use that to directly extract and load only those tables that belong to the font on that index? Do I have to decompress each table and load the whole collection as an SFNT? I'm not sure... * I have written a few macros in `sfwoff2.c' to help with basic calculation and to allow implicit variables. Should I `#undef' these macros towards the end? * Is the code, in general, acceptable? Am I using the FT_* typedefs and macros correctly? Thank you and have a great week! :-) -- Nikhil ___ Freetype-devel mailing list Freetype-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/freetype-devel
