Re: Benefits of Cairo backend over Ghostscript for PDF

[Han-Wen Nienhuys]

On Sat, Mar 18, 2023 at 1:15 PM Jean Abou Samra  wrote:

> > It is possible to avoid this waste by using a
> > graphics package implemented in pure TEX (such
> > as TikZ) or using METAPOST (for which there is
> > special support in dvips, dvipdfm(x) and pdfTEX to
> > avoid font and glyph duplication). The package ps-
> > frag doesn’t suffer from this problem either if the
> > EPS files don’t contain any fonts embedded.
> > *There is no need to pay attention to this tweak,
> > because pdfsizeopt.py unifies font subsets.*
>
> I haven't tried it and don't want to (this topic is too emotional for me 
> right now), but if anyone is interested in reducing the size of doc PDFs when 
> generated with the Cairo backend, this tool could be worth exploring.


See
https://lists.gnu.org/archive/html/lilypond-devel/2023-01/msg00094.html

You could only use this tool if you did some serious refactoring of
the Cairo PDF generation.


-- 
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen

Re: Benefits of Cairo backend over Ghostscript for PDF

[Jean Abou Samra]

(moving to the devel list from the user list)

Le samedi 18 mars 2023 à 05:08 +, Werner LEMBERG a écrit :
> ```
> 
> > I can see that the size of the Cairo-generated PDF is around 3 times
> > bigger compared to the same document generated with Ghostscript?  Is
> > there is way to reduce the size of the Cairo-generated PDF?
> 
> You might try `pdfsizeopt`.
> 
>   [https://github.com/pts/pdfsizeopt](https://github.com/pts/pdfsizeopt)


Interesting. On 
[https://github.com/pts/pdfsizeopt/releases/download/docs-v1/pts_pdfsizeopt2009.psom.pdf](https://github.com/pts/pdfsizeopt/releases/download/docs-v1/pts_pdfsizeopt2009.psom.pdf),
 I read (emphasis mine):


> For glyphs coming from external sources such as the
> included PostScript and PDF graphics, the driver
> is usually not smart enough to recognize the fonts
> already embedded, and unify them with the fonts in
> the main document. Let’s suppose that the docu-
> ment contains included graphics with text captions,
> each graphics source PostScript or PDF having the
> font subsets embedded. No matter dvips, dvipdfm(x)
> or pdfTEX is the driver, it will not be smart enough
> to unify these subsets to a single font. Thus space
> would be wasted in the finally PDF file containing
> multiple subsets of the same font, possibly storing
> duplicate versions of some glyphs.
> It is possible to avoid this waste by using a
> graphics package implemented in pure TEX (such
> as TikZ) or using METAPOST (for which there is
> special support in dvips, dvipdfm(x) and pdfTEX to
> avoid font and glyph duplication). The package ps-
> frag doesn’t suffer from this problem either if the
> EPS files don’t contain any fonts embedded.
> *There is no need to pay attention to this tweak,
> because pdfsizeopt.py unifies font subsets.*

I haven't tried it and don't want to (this topic is too emotional for me right 
now), but if anyone is interested in reducing the size of doc PDFs when 
generated with the Cairo backend, this tool could be worth exploring.


signature.asc
Description: This is a digitally signed message part

Re: Missing items to make Cairo ready

[Jonas Hahnfeld via Discussions on LilyPond development]

On Sun, 2023-01-08 at 12:11 +0100, Han-Wen Nienhuys wrote:
> On Sat, Jan 7, 2023 at 10:16 PM Jonas Hahnfeld  wrote:
> > > > Furthermore, I'm not a fan of recommending two different ways of
> > > > creating PDFs to users (once directly via Cairo and once with ps2pdf),
> > > > unless we really, really have to.
> > > 
> > > We don't really, really have to, but the advantage of dropping \epsfile
> > > and \postscript isn't big either, as their Cairo implementation is not
> > > complicated and can largely share code with the implementation of
> > > other image formats. It may also be useful for people who still use
> > > the PS/EPS output directly and not PDF output (there probably aren't
> > > many for PS, but EPS might be a bit more common?).
> > 
> > That's not really my point; if we keep markup commands that only work
> > via this very specific ps2pdf conversion, we have to guarantee that
> > users get the same visual output as direct PDF output. Do we want to
> > support this? Does Cairo guarantee this for all possible cases that we
> > run into? I highly doubt this is a good rabbit hole to go into...
> > 
> > If we don't do that the error text of \epsfile and \postscript in the
> > default PDF mode has to be "please use this ps2pdf conversion (which we
> > don't even ship with our binaries), and by the way, your PDF will look
> > different". That sounds horrible.
> 
> The discussion was about deprecating \postscript and/or \eps-file.
> There are 2 different ways to understand "deprecate"
> 
> 1. "We are going to remove this command shortly; use the following
> instructions to migrate to the supported XYZ command"
> 
> 2. "We recommend against using this command in new material; use XYZ instead."
> 
> I am arguing against definition 1. If we remove a command, especially
> an open-ended one like \postscript, we are adding a new roadblock to
> users that want to use the latest version of LilyPond, either because
> they want better typography or because they need support for newer
> platforms (eg. windows 64-bit, Apple silicon). The cost of adding this
> roadblock is high for users, while its benefit (not having an ugly
> error message in our code base) is small.

Yes, and I want definition 1: deprecate in 2.26, remove in 3.0 (in
acknowledging the cost). Because from my point of view, the "cost" is
not just the error message, but the entire approach of recommending to
export PS from Cairo and then use Ghostscript to get an inferior PDF.
Which I would argue is entire roadblock itself, as you call it.


signature.asc
Description: This is a digitally signed message part

Re: Missing items to make Cairo ready

[Han-Wen Nienhuys]

On Sat, Jan 7, 2023 at 10:16 PM Jonas Hahnfeld  wrote:
> > > Furthermore, I'm not a fan of recommending two different ways of
> > > creating PDFs to users (once directly via Cairo and once with ps2pdf),
> > > unless we really, really have to.
> >
> > We don't really, really have to, but the advantage of dropping \epsfile
> > and \postscript isn't big either, as their Cairo implementation is not
> > complicated and can largely share code with the implementation of
> > other image formats. It may also be useful for people who still use
> > the PS/EPS output directly and not PDF output (there probably aren't
> > many for PS, but EPS might be a bit more common?).
>
> That's not really my point; if we keep markup commands that only work
> via this very specific ps2pdf conversion, we have to guarantee that
> users get the same visual output as direct PDF output. Do we want to
> support this? Does Cairo guarantee this for all possible cases that we
> run into? I highly doubt this is a good rabbit hole to go into...
>
> If we don't do that the error text of \epsfile and \postscript in the
> default PDF mode has to be "please use this ps2pdf conversion (which we
> don't even ship with our binaries), and by the way, your PDF will look
> different". That sounds horrible.

The discussion was about deprecating \postscript and/or \eps-file.
There are 2 different ways to understand "deprecate"

1. "We are going to remove this command shortly; use the following
instructions to migrate to the supported XYZ command"

2. "We recommend against using this command in new material; use XYZ instead."

I am arguing against definition 1. If we remove a command, especially
an open-ended one like \postscript, we are adding a new roadblock to
users that want to use the latest version of LilyPond, either because
they want better typography or because they need support for newer
platforms (eg. windows 64-bit, Apple silicon). The cost of adding this
roadblock is high for users, while its benefit (not having an ugly
error message in our code base) is small.

The \postscript command is already a rabbit hole, because

* it offers access to undocumented postscript commands
(staff-line-thickness, draw_round_box, etc.).
* it doesn't work with SVG output.
* it's also a "specialist" command as writing postscript isn't for the
faint of heart.

However removing the rabbit-hole doesn't actually help users that have
already used it in their .ly files.

I am arguing that we should do what is in definition 2  (but I usually
don't call this deprecate)

-- 
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 07/01/2023 à 22:58, Jonas Hahnfeld a écrit :

That's only the EPS file; but for all the rest, is Cairo -> PS ->
ps2pdf -> PDF identical to Cairo -> PDF?



Hm, ok, true, there is one difference: you will not get
a PDF outline for a TOC or PDF metadata.



OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Luca Fascione]

On Sat, Jan 7, 2023 at 11:23 PM Jean Abou Samra  wrote:

> All I said in the quote is that it does not take a lot of code in
> LilyPond to support embedding EPS files into PS/EPS output in the
> Cairo backend.
>

Forgive me Jean, I thought you were talking about the Cairo/PDF backend.
In that scenario I had understood from the previous messages that the idea
is to render
the EPS/postscript content into rasterized graphics, and I thought this is
what you were referring to here as well.
My mistake.

I don't know why you think there is any raster rendering
>

I thought it was your proposal to implement \epsfile by using gs to render
EPS into a PNG,
and embed that into the resulting PDF.
I seem to have misunderstood something you said.

L


-- 
Luca Fascione

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 07/01/2023 à 23:04, Luca Fascione a écrit :
On Sat, Jan 7, 2023 at 10:06 PM Jean Abou Samra  
wrote:


the advantage of dropping \epsfile
and \postscript isn't big either, as their Cairo implementation is not
complicated and can largely share code with the implementation of
other image formats


Hang on. I don't think this is correct: I had understood that on the 
postscript backend
the postscript in the arguments to the two commands above goes 
verbatim into the output file, doesn't it?




More or less, yes, but how does this relate to or contradict
what I wrote above in any way?

All I said in the quote is that it does not take a lot of code in
LilyPond to support embedding EPS files into PS/EPS output in the
Cairo backend.



In particular this means the PDF contains vector images when the input 
is things like paths and such stuff,
or scalable fonts, possibly together with their definitions when 
that's the case with the input.
Comparing this to Cairo where an external renderer is invoked to turn 
them into raster images and

then embeds those into a PDF, which would make it all non-scalable.




Current LilyPond is

LilyPond      ->   PS    ->   PDF
    PS backend   Ghostscript

The new route to go from Cairo to PDF if you really need
EPS is

LilyPond  ->  Cairo    ->  PS   -> PDF
  Cairo backend   Cairo    Ghostscript

The PS -> PDF step is done in the same way in both cases.
I don't know why you think there is any raster rendering.



OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Luca Fascione]

On Sat, Jan 7, 2023 at 10:06 PM Jean Abou Samra  wrote:

> the advantage of dropping \epsfile
> and \postscript isn't big either, as their Cairo implementation is not
> complicated and can largely share code with the implementation of
> other image formats
>

Hang on. I don't think this is correct: I had understood that on the
postscript backend
the postscript in the arguments to the two commands above goes verbatim
into the output file, doesn't it?
In particular this means the PDF contains vector images when the input is
things like paths and such stuff,
or scalable fonts, possibly together with their definitions when that's the
case with the input.
Comparing this to Cairo where an external renderer is invoked to turn them
into raster images and
then embeds those into a PDF, which would make it all non-scalable.

This is all assuming that \postscript is not passing through actual code
that modifies the state of the postscript interpreter
which the new system can't even do (like you could have a \postscript
command to set up a dictionary, later in your file another to
set up some state, and further to read such state and generate text, for
example you could generate page numbers like this if you wanted, or
maybe back references (not that it would be a smart thing to do, but you
could). I don't quite see how running this through ghostscript a bit at a
time
would achieve the right result.

So it seems to me \postscript and \epsfile can only do exactly what they
currently do if the emitted file is postscript.

In saying this, I think that them not working in other output modes is
just fine (as long as the use has a way to check what output mode is
currently running).
I'm thinking if we had similar mechanism for direct ingestion of XML in SVG
mode (I do realize it's a much weirder thing to make available in
practice), and equivalently
direct emission of PDF, and each just did something useful in the
corresponding mode, it would be just fine.

The TeX people are in this situation, and it doesn't seem to bother users
much (because it's largely hidden by the packages, arguably, at least in
LaTeX).

L


-- 
Luca Fascione

Re: Missing items to make Cairo ready

[Jonas Hahnfeld via Discussions on LilyPond development]

On Sat, 2023-01-07 at 22:25 +0100, Jean Abou Samra wrote:
> Le 07/01/2023 à 22:16, Jonas Hahnfeld a écrit :
> > That's not really my point; if we keep markup commands that only work
> > via this very specific ps2pdf conversion, we have to guarantee that
> > users get the same visual output as direct PDF output. Do we want to
> > support this? Does Cairo guarantee this for all possible cases that we
> > run into? I highly doubt this is a good rabbit hole to go into...
> 
> I don't see an obvious reason for the PDF output to differ noticeably from
> the current output since Cairo doesn't process the EPS code in any way
> but just embeds it in pretty much the same way as the PS backend does,
> and the EPS -> PDF rendering is done by GS in both cases.

That's only the EPS file; but for all the rest, is Cairo -> PS ->
ps2pdf -> PDF identical to Cairo -> PDF?


signature.asc
Description: This is a digitally signed message part

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 07/01/2023 à 22:16, Jonas Hahnfeld a écrit :

That's not really my point; if we keep markup commands that only work
via this very specific ps2pdf conversion, we have to guarantee that
users get the same visual output as direct PDF output. Do we want to
support this? Does Cairo guarantee this for all possible cases that we
run into? I highly doubt this is a good rabbit hole to go into...




I don't see an obvious reason for the PDF output to differ noticeably from
the current output since Cairo doesn't process the EPS code in any way
but just embeds it in pretty much the same way as the PS backend does,
and the EPS -> PDF rendering is done by GS in both cases.



If we don't do that the error text of \epsfile and \postscript in the
default PDF mode has to be "please use this ps2pdf conversion (which we
don't even ship with our binaries), and by the way, your PDF will look
different". That sounds horrible.



We don't need to mention ps2pdf at all. Just "You may still use
\epsfile when generating PS / EPS output. For other formats,
images must be in PNG[/SVG/JPEG possibly] formats." I assume
someone who really wants to include EPSes for some reason knows
a bit about what the EPS format is.



OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Jonas Hahnfeld via Discussions on LilyPond development]

On Sat, 2023-01-07 at 22:05 +0100, Jean Abou Samra wrote:
> Le 07/01/2023 à 21:53, Jonas Hahnfeld a écrit :
> > Furthermore, I'm not a fan of recommending two different ways of
> > creating PDFs to users (once directly via Cairo and once with ps2pdf),
> > unless we really, really have to.
> 
> We don't really, really have to, but the advantage of dropping \epsfile
> and \postscript isn't big either, as their Cairo implementation is not
> complicated and can largely share code with the implementation of
> other image formats. It may also be useful for people who still use
> the PS/EPS output directly and not PDF output (there probably aren't
> many for PS, but EPS might be a bit more common?).

That's not really my point; if we keep markup commands that only work
via this very specific ps2pdf conversion, we have to guarantee that
users get the same visual output as direct PDF output. Do we want to
support this? Does Cairo guarantee this for all possible cases that we
run into? I highly doubt this is a good rabbit hole to go into...

If we don't do that the error text of \epsfile and \postscript in the
default PDF mode has to be "please use this ps2pdf conversion (which we
don't even ship with our binaries), and by the way, your PDF will look
different". That sounds horrible.


signature.asc
Description: This is a digitally signed message part

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 07/01/2023 à 21:53, Jonas Hahnfeld a écrit :

Furthermore, I'm not a fan of recommending two different ways of
creating PDFs to users (once directly via Cairo and once with ps2pdf),
unless we really, really have to.



We don't really, really have to, but the advantage of dropping \epsfile
and \postscript isn't big either, as their Cairo implementation is not
complicated and can largely share code with the implementation of
other image formats. It may also be useful for people who still use
the PS/EPS output directly and not PDF output (there probably aren't
many for PS, but EPS might be a bit more common?).



OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Jonas Hahnfeld via Discussions on LilyPond development]

On Fri, 2023-01-06 at 23:19 +0100, Han-Wen Nienhuys wrote:
> On Fri, Jan 6, 2023 at 10:19 AM Jonas Hahnfeld  wrote:
> > 
> > On Wed, 2023-01-04 at 12:52 +0100, Han-Wen Nienhuys wrote:
> > > Regarding versioning: the 1.x to 2.x transition was motivated by
> > > radical syntax changes that necessitated converting and 'manually'
> > > verifying the .ly files. Since Cairo vs. Ghostscript doesn't affect
> > > the semantics of .ly files, I think we can continue the 2.x version
> > > number. As a practical example, page layout was introduced in 2.4, and
> > > direct to PostScript only became default in 2.6; both changes are much
> > > more invasive than what we are discussing here.
> > 
> > Regardless of what has been done in prior versions, it seems to me the
> > cleanest solution still is to remove a number of markup commands that
> > we cannot or do not want to support with Cairo. We know some are used
> > in existing libraries and scores, so this constitutes a breaking
> > change. What exactly is your argument for *not* going to version 3.x in
> > that case?
> 
> I don't think there is value in removing markup commands. When we drop
> the PS backend, folks that use \postscript or \epsfile can do
> 
>   lilypond --ps foo.ly && ps2pdf foo.ps
> 
> rather than
> 
>   lilypond foo.ly
> 
> and have their scores mostly work. We don't need to remove support for
> this, ever. Since this doesn't break backward compatibility, I don't
> think we need a major version bump.

For me, this meets *exactly* the definition of a backward
incompatibility: Something that used to work (just running lilypond
foo.ly to get a PDF) doesn't give the expected result anymore.
Furthermore, I'm not a fan of recommending two different ways of
creating PDFs to users (once directly via Cairo and once with ps2pdf),
unless we really, really have to.

I can also buy Werner's arguments about a fundamental change, and I
would use the jump to advertise as such, but for me the purely
objective reason are backwards incompatibilities that we *will* have
when going to Cairo.


signature.asc
Description: This is a digitally signed message part

Re: Missing items to make Cairo ready

[Werner LEMBERG]

>> What exactly is your argument for *not* going to version 3.x in
>> that case?
> 
> [...]  Since this doesn't break backward compatibility, I don't
> think we need a major version bump.

IMHO, a major version bump should also be used if something changes
fundamentally, regardless whether it is backward compatible of not.
The introduction of the Cairo backend as the default is such an event.
LilyPond is not a library, we are thus not bound to reflect the DLL
version number in some way.  Also, many other software products do the
same, including other GNU projects – a new major version number is
kind of an advertisement.

At the current maturity level of LilyPond I don't see that we are ever
going to change the syntax in such a radical way that it would deserve
a major version change.  Additionally, my gut feeling says that the
minor version number is already uncomfortably high.

The alternative is to drop the major number completely, as was done
for Emacs, Firefox, or the Linux kernel, to name some well-known
projects.  However, then the major version number gets uncomfortably
high...  But maybe it's only me who thinks that :-)


Werner

Re: Missing items to make Cairo ready

[Han-Wen Nienhuys]

On Fri, Jan 6, 2023 at 10:19 AM Jonas Hahnfeld  wrote:
>
> On Wed, 2023-01-04 at 12:52 +0100, Han-Wen Nienhuys wrote:
> > Regarding versioning: the 1.x to 2.x transition was motivated by
> > radical syntax changes that necessitated converting and 'manually'
> > verifying the .ly files. Since Cairo vs. Ghostscript doesn't affect
> > the semantics of .ly files, I think we can continue the 2.x version
> > number. As a practical example, page layout was introduced in 2.4, and
> > direct to PostScript only became default in 2.6; both changes are much
> > more invasive than what we are discussing here.
>
> Regardless of what has been done in prior versions, it seems to me the
> cleanest solution still is to remove a number of markup commands that
> we cannot or do not want to support with Cairo. We know some are used
> in existing libraries and scores, so this constitutes a breaking
> change. What exactly is your argument for *not* going to version 3.x in
> that case?

I don't think there is value in removing markup commands. When we drop
the PS backend, folks that use \postscript or \epsfile can do

  lilypond --ps foo.ly && ps2pdf foo.ps

rather than

  lilypond foo.ly

and have their scores mostly work. We don't need to remove support for
this, ever. Since this doesn't break backward compatibility, I don't
think we need a major version bump.

-- 
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen

Re: Missing items to make Cairo ready

[Marnen Laibow-Koser]

On Fri, Jan 6, 2023 at 4:34 PM Marnen Laibow-Koser 
wrote:
[…]

> BTW, what is the current status of Mac .app builds of LilyPond? I haven’t
> seen any new builds available since the last one that I created, but maybe
> I’m looking in the wrong place.
>

I see there are builds available on the website that weren’t there when I
last checked. I’m looking forward to trying them out!


>
>>
>> Thank you,
>> Jean
>>
>> Best,
> --
> Marnen Laibow-Koser
> mar...@marnen.org
> http://www.marnen.org
>
> Sent from Gmail Mobile
>
-- 
Marnen Laibow-Koser
mar...@marnen.org
http://www.marnen.org

Sent from Gmail Mobile

Re: Missing items to make Cairo ready

[Marnen Laibow-Koser]

On Fri, Jan 6, 2023 at 8:23 AM Jean Abou Samra  wrote:

> Le 06/01/2023 à 10:13, Jonas Hahnfeld a écrit :
> > On Fri, 2023-01-06 at 08:48 +0100, Jean Abou Samra wrote:
> >> Jonas, is there a possibility to have access to the MacStadium
> >> node, or do you have other advice on testing this on macOS?
> >> I'd like to check it also works there before potentially starting
> >> a discussion on requiring librsvg.
> > I'd love to create you an account on the MacStadium node (or any
> > developer that is interested), but sadly I have no permission to do so
> > and only have an unprivileged account myself. We can try contacting
> > Marnen Laibow-Koser and see if he is willing to create one for you.
>
>
>
> Hi Marnen,
>
> We are discussing a possible dependency addition to LilyPond,
> and I would like to experiment with building this new dependency
> on macOS. Would it be possible for me to have access to the
> MacStadium node for this?


Of course. I’ll set up an account for you (I’ll send you login info
separately as soon as I do so).

BTW, what is the current status of Mac .app builds of LilyPond? I haven’t
seen any new builds available since the last one that I created, but maybe
I’m looking in the wrong place.


>
> Thank you,
> Jean
>
> Best,
-- 
Marnen Laibow-Koser
mar...@marnen.org
http://www.marnen.org

Sent from Gmail Mobile

Re: Missing items to make Cairo ready

[Jean Abou Samra]

   De : Han-Wen Nienhuys <[1]hanw...@gmail.com>

   À : Jean Abou Samra <[2]j...@abou-samra.fr>

   CC : Jonas Hahnfeld <[3]hah...@hahnjo.de>, lilypond-devel
   <[4]lilypond-devel@gnu.org>

   Date : 06/01/2023 18:53 CET

   Sujet : Re: Missing items to make Cairo ready





   On Sun, Jan 1, 2023 at 12:36 PM Jean Abou Samra <[5]j...@abou-samra.fr>
   wrote:

   >

   Le 30/12/2022 à 13:08, Jean Abou Samra a écrit :

   which means figuring out how to do PNGs via the default PS

   backend and GS.

   >

 I looked a bit at this.

   It's not insurmountable, *but*, alpha transparency is not going

   to work.

   transparency doesn't work today either in the PS backend, the

   stencil-expressions.ly regtest looks different in Cairo and the

   classic backend. See commit 103ca4c38061754f612880bcc7c29b4fd5acb8f6.




   Yes. However, you don’t get a transparent
   color without having explicitly asked for it. On the
   other hand, if we have this discrepancy that the PS
   backend draws PNG images on a white background
   while the Cairo backend makes the transparent parts
   really transparent, we have one more incompatibility
   that people will need to pay attention to when switching
   to Cairo. That’s why I prefer making all backends default
   to a white background.

References

   1. mailto:hanw...@gmail.com
   2. mailto:j...@abou-samra.fr
   3. mailto:hah...@hahnjo.de
   4. mailto:lilypond-devel@gnu.org
   5. mailto:j...@abou-samra.fr

Re: Missing items to make Cairo ready

[Han-Wen Nienhuys]

On Sun, Jan 1, 2023 at 12:36 PM Jean Abou Samra  wrote:
>
> Le 30/12/2022 à 13:08, Jean Abou Samra a écrit :
> > which means figuring out how to do PNGs via the default PS
> > backend and GS.
>
>
> I looked a bit at this.
>
> It's not insurmountable, *but*, alpha transparency is not going
> to work.

transparency doesn't work today either in the PS backend, the
stencil-expressions.ly regtest looks different in Cairo and the
classic backend. See commit 103ca4c38061754f612880bcc7c29b4fd5acb8f6.

-- 
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen

Re: Missing items to make Cairo ready

[Jonas Hahnfeld via Discussions on LilyPond development]

On Fri, 2023-01-06 at 14:41 +0100, Thomas Morley wrote:
> Am Fr., 6. Jan. 2023 um 14:26 Uhr schrieb Jonas Hahnfeld via
> Discussions on LilyPond development :
> > 
> > On Fri, 2023-01-06 at 14:21 +0100, Jean Abou Samra wrote:
> > > Le 06/01/2023 à 10:19, Jonas Hahnfeld a écrit :
> 
> > > I don't see any markup commands other than \postscript
> > > and \epsfile that we cannot fully support in Cairo.
> > 
> > Any number bigger than zero represents a "breaking change" for me, and
> > it's not the sort of change that can be fixed by convert-ly or even
> > manually because there is no (general) equivalent.
> 
> Imho, a good point to look at what users do with \postscript is the LSR.
> 
> Currently 20 snippets mention 'postscript'. I'm pretty sure most of
> them can be modified to use \path etc.
> Apart from https://lsr.di.unimi.it/LSR/Item?id=1060
> This one is nice, though.
> 
> Anyway, I'd offer to modify said snippets during the 2.25. circle.
> Maybe it helps deprecating \postscript ?

Yes, this has been my proposal all along: Deprecate it for one stable
release, and then we can remove it with the major version 3.0 along
with Cairo becoming the default.


signature.asc
Description: This is a digitally signed message part

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 06/01/2023 à 14:41, Thomas Morley a écrit :

Imho, a good point to look at what users do with \postscript is the LSR.
Currently 20 snippets mention 'postscript'. I'm pretty sure most of
them can be modified to use \path etc.
Apart from https://lsr.di.unimi.it/LSR/Item?id=1060
This one is nice, though.



Hm, I have to think how easy it is to add proper support for color 
gradients...




Anyway, I'd offer to modify said snippets during the 2.25. circle.



Yes, thank you for proposing, that would help a lot!


Best,
Jean



OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Thomas Morley]

Am Fr., 6. Jan. 2023 um 14:26 Uhr schrieb Jonas Hahnfeld via
Discussions on LilyPond development :
>
> On Fri, 2023-01-06 at 14:21 +0100, Jean Abou Samra wrote:
> > Le 06/01/2023 à 10:19, Jonas Hahnfeld a écrit :

> > I don't see any markup commands other than \postscript
> > and \epsfile that we cannot fully support in Cairo.
>
> Any number bigger than zero represents a "breaking change" for me, and
> it's not the sort of change that can be fixed by convert-ly or even
> manually because there is no (general) equivalent.

Imho, a good point to look at what users do with \postscript is the LSR.

Currently 20 snippets mention 'postscript'. I'm pretty sure most of
them can be modified to use \path etc.
Apart from https://lsr.di.unimi.it/LSR/Item?id=1060
This one is nice, though.

Anyway, I'd offer to modify said snippets during the 2.25. circle.
Maybe it helps deprecating \postscript ?


Only one lsr-file uses \epsfile.

Cheers,
  Harm

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 06/01/2023 à 14:26, Jonas Hahnfeld a écrit :

On Fri, 2023-01-06 at 14:21 +0100, Jean Abou Samra wrote:

Le 06/01/2023 à 10:19, Jonas Hahnfeld a écrit :

Regardless of what has been done in prior versions, it seems to me the
cleanest solution still is to remove a number of markup commands that
we cannot or do not want to support with Cairo.

I don't see any markup commands other than \postscript
and \epsfile that we cannot fully support in Cairo.

Any number bigger than zero represents a "breaking change" for me, and
it's not the sort of change that can be fixed by convert-ly or even
manually because there is no (general) equivalent.




Oh, I was not trying to comment on whether we should bump the major
version (I have no opinion on that), just on which deprecations
actually have to be made.




The situation for these is that they are only partially
supported, they only work for PS/EPS output. I see no
point in removing them entirely, we can just deprecate
calling them for non-PS/EPS output.

I don't understand the desire to keep something half-working...




No strong opinion here, but it gives an escape hatch to obtain PDFs
with embedded EPSes, by generating PS output and converting
to PDF yourself, which may ease the transition for some people.




OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Jonas Hahnfeld via Discussions on LilyPond development]

On Fri, 2023-01-06 at 14:21 +0100, Jean Abou Samra wrote:
> Le 06/01/2023 à 10:19, Jonas Hahnfeld a écrit :
> > Regardless of what has been done in prior versions, it seems to me the
> > cleanest solution still is to remove a number of markup commands that
> > we cannot or do not want to support with Cairo.
> 
> I don't see any markup commands other than \postscript
> and \epsfile that we cannot fully support in Cairo.

Any number bigger than zero represents a "breaking change" for me, and
it's not the sort of change that can be fixed by convert-ly or even
manually because there is no (general) equivalent.

> The situation for these is that they are only partially
> supported, they only work for PS/EPS output. I see no
> point in removing them entirely, we can just deprecate
> calling them for non-PS/EPS output.

I don't understand the desire to keep something half-working...

> However, I think
> the time to discuss this is not right now, but at least
> when we have taken a decision on if/when we want to
> support SVG images via librsvg.

Ok.


signature.asc
Description: This is a digitally signed message part

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 06/01/2023 à 10:13, Jonas Hahnfeld a écrit :

On Fri, 2023-01-06 at 08:48 +0100, Jean Abou Samra wrote:

Jonas, is there a possibility to have access to the MacStadium
node, or do you have other advice on testing this on macOS?
I'd like to check it also works there before potentially starting
a discussion on requiring librsvg.

I'd love to create you an account on the MacStadium node (or any
developer that is interested), but sadly I have no permission to do so
and only have an unprivileged account myself. We can try contacting
Marnen Laibow-Koser and see if he is willing to create one for you.




Hi Marnen,

We are discussing a possible dependency addition to LilyPond,
and I would like to experiment with building this new dependency
on macOS. Would it be possible for me to have access to the
MacStadium node for this?

Thank you,
Jean



OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 06/01/2023 à 10:19, Jonas Hahnfeld a écrit :

Regardless of what has been done in prior versions, it seems to me the
cleanest solution still is to remove a number of markup commands that
we cannot or do not want to support with Cairo.



I don't see any markup commands other than \postscript
and \epsfile that we cannot fully support in Cairo.
The situation for these is that they are only partially
supported, they only work for PS/EPS output. I see no
point in removing them entirely, we can just deprecate
calling them for non-PS/EPS output. However, I think
the time to discuss this is not right now, but at least
when we have taken a decision on if/when we want to
support SVG images via librsvg.




OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Jonas Hahnfeld via Discussions on LilyPond development]

On Thu, 2023-01-05 at 23:30 +0100, Jonas Hahnfeld via Discussions on
LilyPond development wrote:
> On Thu, 2023-01-05 at 13:24 +, Werner LEMBERG wrote:
> > > 
> > > IMO, working with a 35mb user manual isn't materially different
> > > from working with a 10mb user manual.  Both take a while to
> > > download.
> > 
> > Indeed, but the manuals as a whole, in all languages, get also
> > distributed, and there it *does* make a significant difference IMHO:
> > Right now, the PDFs in `lilypond-2.24.0-documentation.tar.xz` (which
> > has a size of 170MByte) need 144MByte in total (uncompressed).
> > Multiply the latter by four...
> 
> Let's look at some concrete and objective numbers here instead of just
> extrapolating - notation.pdf and also collated-files.pdf with all
> regression tests are kind of the worst case scenario for the inclusion
> of many tiny PDFs and font subsetting.
> 
> When building with gs-9.55.0 and extractpdfmark, notation.pdf and
> collated-files.pdf are 6.3MB and 4.9MB. With the Cairo backend
> (admittedly post-processed with gs-10.0.0; will have to re-run with the
> older gs version tomorrow), this increases to 14MB and 17MB - a factor
> 2.2x and 3.5x.
> For the totality of the (uncompressed) out-www/offline-root, this means
> an increase from 825MB to 898MB; when tar'ing only this directory with
> xz, the size grows from 130MB to 179MB. (Please keep in mind that our
> documentation tarball contains more than that, and is also built in a
> different environment, so numbers may and will vary).

Using cairo with gs-9.55.0 for post-processing (and with working
links), the sizes increased some more: notation.pdf is now 19M (a
factor 3x) and collated-files.pdf is 25M (a factor 5x). The size of the
uncompressed out-www/offline-root jumps to 952MB and tar'ed with xz it
is now 194MB - almost a factor 1.5x.


signature.asc
Description: This is a digitally signed message part

Re: Missing items to make Cairo ready

[Jonas Hahnfeld via Discussions on LilyPond development]

On Wed, 2023-01-04 at 12:52 +0100, Han-Wen Nienhuys wrote:
> Regarding versioning: the 1.x to 2.x transition was motivated by
> radical syntax changes that necessitated converting and 'manually'
> verifying the .ly files. Since Cairo vs. Ghostscript doesn't affect
> the semantics of .ly files, I think we can continue the 2.x version
> number. As a practical example, page layout was introduced in 2.4, and
> direct to PostScript only became default in 2.6; both changes are much
> more invasive than what we are discussing here.

Regardless of what has been done in prior versions, it seems to me the
cleanest solution still is to remove a number of markup commands that
we cannot or do not want to support with Cairo. We know some are used
in existing libraries and scores, so this constitutes a breaking
change. What exactly is your argument for *not* going to version 3.x in
that case?


signature.asc
Description: This is a digitally signed message part

Re: Missing items to make Cairo ready

[Jonas Hahnfeld via Discussions on LilyPond development]

On Fri, 2023-01-06 at 08:48 +0100, Jean Abou Samra wrote:
> Le 06/01/2023 à 03:19, Jean Abou Samra a écrit :
> > Le 04/01/2023 à 15:50, Jonas Hahnfeld a écrit :
> > > On the other hand, librsvg is written in Rust where I have no
> > > experience how practical it actually is to integrate into our binaries
> > > on all platforms.
> > 
> > To my surprise, I was able to get librsvg to build in 
> > release/binaries/ this evening, with the three dependencies it pulls in
> > (libxml, libjpeg and gdk-pixbuf). I'll try cross-compilation next, we'll
> > see how far that goes ...
> 
> I have to say this went pretty smoothly. Took some work, but I was
> expecting far worse. After a few hours, I got Windows binaries that
> run just fine under Wine.
> 
> Jonas, is there a possibility to have access to the MacStadium
> node, or do you have other advice on testing this on macOS?
> I'd like to check it also works there before potentially starting
> a discussion on requiring librsvg.

I'd love to create you an account on the MacStadium node (or any
developer that is interested), but sadly I have no permission to do so
and only have an unprivileged account myself. We can try contacting
Marnen Laibow-Koser and see if he is willing to create one for you.


signature.asc
Description: This is a digitally signed message part

Re: Missing items to make Cairo ready

[Jonas Hahnfeld via Discussions on LilyPond development]

On Thu, 2023-01-05 at 23:55 +0100, Jean Abou Samra wrote:
> Le 05/01/2023 à 23:30, Jonas Hahnfeld a écrit :
> > What I find worrying in this discussion is that proponents of having
> > Cairo sooner than later keep dismissing the size argument, in parts
> > with very strong words, without the willingness to look into it (as far
> > as I understand; or have there been concrete attempts before I raised
> > the point only yesterday?). In my humble opinion, this is not a good
> > attitude for planning and proposing large-scale changes as we are
> > discussing them...
> 
> No, I had not looked into this before yesterday or thought about it,
> and I am not ashamed about it, because the exact purpose of this
> thread in the first place was to ask what the potential blockers
> were, you brought up one, and that is the kind of answer I expected.

In my personal opinion, this stage is even less of a place to make
statements like "Quite frankly, I think it's not going to happen until
the 12th of Never." and "I strongly disagree that PDFs of the same size
as today is a requirement [...]".

> Again, I am not proposing anything. I'm just trying to figure out
> a minimum amount of work that needs to be done before a concrete
> proposal for a switch can even meaningfully be made (before the thread
> largely derailed on the actual details of this not-yet-made proposal,
> which was not really the intent, but anyway).

The original post had this:
> I ask this because we are at an early point in the 2.26 release
> cycle, which could potentially be ideal to get testing for "Cairo by
> default" if it were ready, but it isn't yet.

"early point in the 2.26 release cycle" is basically now, not several
months from now. I'm sorry, but I see no other interpretation than your
intent to make a proposal into that direction very, very soon.

> I tend to shrug about size issues because I view them as the least
> of my worries compared to PDF / SVG features, which everybody agrees
> are critical to get Cairo ready for being a default, and I don't have
> high hopes of doing anything about them anyway.

Can you please stop making such statements? Complex problems aren't
solved within one day. Even if it's not obvious now, I am confident
that there are enough people around here that somebody will find a
solution eventually. If we try, that is, and not just shrug about it.


signature.asc
Description: This is a digitally signed message part

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 06/01/2023 à 03:19, Jean Abou Samra a écrit :

Le 04/01/2023 à 15:50, Jonas Hahnfeld a écrit :

On the other hand, librsvg is written in Rust where I have no
experience how practical it actually is to integrate into our binaries
on all platforms.




To my surprise, I was able to get librsvg to build in 
release/binaries/ this

evening, with the three dependencies it pulls in (libxml, libjpeg and
gdk-pixbuf). I'll try cross-compilation next, we'll see how far that 
goes ...







I have to say this went pretty smoothly. Took some work, but I was
expecting far worse. After a few hours, I got Windows binaries that
run just fine under Wine.

Jonas, is there a possibility to have access to the MacStadium
node, or do you have other advice on testing this on macOS?
I'd like to check it also works there before potentially starting
a discussion on requiring librsvg.



OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 04/01/2023 à 15:50, Jonas Hahnfeld a écrit :

On the other hand, librsvg is written in Rust where I have no
experience how practical it actually is to integrate into our binaries
on all platforms.




To my surprise, I was able to get librsvg to build in release/binaries/ this
evening, with the three dependencies it pulls in (libxml, libjpeg and
gdk-pixbuf). I'll try cross-compilation next, we'll see how far that 
goes ...




OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 05/01/2023 à 23:30, Jonas Hahnfeld a écrit :

What I find worrying in this discussion is that proponents of having
Cairo sooner than later keep dismissing the size argument, in parts
with very strong words, without the willingness to look into it (as far
as I understand; or have there been concrete attempts before I raised
the point only yesterday?). In my humble opinion, this is not a good
attitude for planning and proposing large-scale changes as we are
discussing them...




No, I had not looked into this before yesterday or thought about it,
and I am not ashamed about it, because the exact purpose of this
thread in the first place was to ask what the potential blockers
were, you brought up one, and that is the kind of answer I expected.

Again, I am not proposing anything. I'm just trying to figure out
a minimum amount of work that needs to be done before a concrete
proposal for a switch can even meaningfully be made (before the thread
largely derailed on the actual details of this not-yet-made proposal,
which was not really the intent, but anyway).

I tend to shrug about size issues because I view them as the least
of my worries compared to PDF / SVG features, which everybody agrees
are critical to get Cairo ready for being a default, and I don't have
high hopes of doing anything about them anyway.



OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Karlin High]

On 1/5/2023 4:30 PM, Jonas Hahnfeld wrote:

What I find worrying in this discussion is that proponents of having
Cairo sooner than later keep dismissing the size argument


Concern granted. I do appreciate optimal PDFs.

How much effort should be expended for what level of optimization, 
Cairo's costs in PDF optimization vs benefits in SVG output, I will not 
have the expertise to advance those discussions very much.


Thanks for the reminder to minimize dismissiveness.
--
Karlin High
Missouri, USA

Re: Missing items to make Cairo ready

[Han-Wen Nienhuys]

On Thu, Jan 5, 2023 at 2:24 PM Werner LEMBERG  wrote:
>
>
> >> The only thing I would like to convince the Cairo people is to add
> >> a mode to produce PDFs with font references instead of embedding –
> >> and subsetting – fonts.  My Cairo knowledge is zero; maybe this is
> >> already possible?
> >
> > This makes no sense to me as a Cairo feature. Such PDF snippets
> > would need custom post processing to assemble into the full
> > document.
>
> The general problem (i.e., the inclusion of multiple PDF images into a
> larger PDF document) is not unique to LilyPond.  My idea is that there
> exist special tools (for example, Ghostscript's 'pdfwrite' device or
> `pdfsizeopt`) to produce size-optimized PDFs.[*] However, these tools
> can only do their optimization job if the necessary preliminaries are
> fulfilled.  If the PDF images contain subsetted fonts, it doesn't work
> most of the time: You either need PDFs with complete fonts (i.e., not
> subsetted), or PDFs with references to fonts.  I've now looked up the
> Cairo manual, and it doesn't offer any control for that.
>
> > If we do custom post processing, we might as well postprocess the
> > final PDF directly to make all snippets point to a single music font
> > object?
>
> Font subsetting effectively prevents such post-processing since too
> much information gets stripped off during this process.  For the
> special case of Emmentaler fonts it might work because LilyPond knows
> more about these fonts than for others, but not in the general case.

I played around on essay.pdf with pdfcpu. It looks like Cairo strips
the encoding information when the snippet PDFs are generated, and then
creates a subsetted font containing /uni0001, /uni0002 in glyph slots
1, 2, .. in the font. I was hoping we could replace the font reference
within the snippet in a postprocessing step, but this is obviously
impossible, as each snippet has its own encoding. It also means that
the Cairo feature you propose would add a completely new way of
outputting glyphs/fonts, with associated costs in coding, testing etc.
>From the perspective of Cairo development, it seems like a tall order.

> > IMO, working with a 35mb user manual isn't materially different from
> > working with a 10mb user manual.  Both take a while to download.
>
> Indeed, but the manuals as a whole, in all languages, get also
> distributed, and there it *does* make a significant difference IMHO:
> Right now, the PDFs in `lilypond-2.24.0-documentation.tar.xz` (which
> has a size of 170MByte) need 144MByte in total (uncompressed).
> Multiply the latter by four...

Do people actually download this a lot? Unfortunately Gitlab doesn't
provide this data
(https://gitlab.com/gitlab-org/gitlab/-/issues/327317).  I looked on
lilypond.org as well, but we only have 2 weeks of data and the doc
tarballs there are outdated (there were no downloads.)

-- 
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen

Re: Missing items to make Cairo ready

[Jonas Hahnfeld via Discussions on LilyPond development]

On Thu, 2023-01-05 at 13:24 +, Werner LEMBERG wrote:
> > 
> > IMO, working with a 35mb user manual isn't materially different
> > from working with a 10mb user manual.  Both take a while to
> > download.
> 
> Indeed, but the manuals as a whole, in all languages, get also
> distributed, and there it *does* make a significant difference IMHO:
> Right now, the PDFs in `lilypond-2.24.0-documentation.tar.xz` (which
> has a size of 170MByte) need 144MByte in total (uncompressed).
> Multiply the latter by four...

Let's look at some concrete and objective numbers here instead of just
extrapolating - notation.pdf and also collated-files.pdf with all
regression tests are kind of the worst case scenario for the inclusion
of many tiny PDFs and font subsetting.

When building with gs-9.55.0 and extractpdfmark, notation.pdf and
collated-files.pdf are 6.3MB and 4.9MB. With the Cairo backend
(admittedly post-processed with gs-10.0.0; will have to re-run with the
older gs version tomorrow), this increases to 14MB and 17MB - a factor
2.2x and 3.5x.
For the totality of the (uncompressed) out-www/offline-root, this means
an increase from 825MB to 898MB; when tar'ing only this directory with
xz, the size grows from 130MB to 179MB. (Please keep in mind that our
documentation tarball contains more than that, and is also built in a
different environment, so numbers may and will vary).

Is this increase measurable? Yes. Is it acceptable? I don't know, but
simply pointing out that it has been worse before the introduction of
extractpdfmark is not a really convincing argument IMHO.

What I find worrying in this discussion is that proponents of having
Cairo sooner than later keep dismissing the size argument, in parts
with very strong words, without the willingness to look into it (as far
as I understand; or have there been concrete attempts before I raised
the point only yesterday?). In my humble opinion, this is not a good
attitude for planning and proposing large-scale changes as we are
discussing them...

Jonas


signature.asc
Description: This is a digitally signed message part

Ghostscript and new PDF interpreter (was: Missing items to make Cairo ready)

[Jonas Hahnfeld via Discussions on LilyPond development]

[ replying on the topic of the new PDF interpreter alone because it
really bothers me that you keep making wrong claims in that regard ]

On Wed, 2023-01-04 at 15:13 +, Werner LEMBERG wrote:
> > With extractpdfmark
> > ===
> > 
> > PS backend:
> > 
> > 4,8M    Documentation/out-www/en/notation.pdf
> 
> FYI: Using a recent GS version with its new PDF backend, the size
> gets larger than 20MByte.  I guess this is a bug, and I will soon
> report this to the GS people so that it gets fixed in the next GS
> version (in March), which will eventually drop the old PDF engine
> completely.
> 
> On the other hand, GS's new PDF engine no longer contains a PS
> interpreter; it is possible that `extractpdfmark` doesn't work
> anymore, and we have to find something new...

This is wrong, or at least the new PDF *interpreter* (which is, in my
understanding, not to be confused with the PDF driver to output PDF) in
itself is not to blame here: The new interpreter was already enabled by
default in Ghostscript 9.56 and this version still works perfectly
fine, yielding small PDF sizes together with extractpdfmark.


signature.asc
Description: This is a digitally signed message part

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 05/01/2023 à 16:14, Werner LEMBERG a écrit :

This is apparently a generation thing: I still can remember using
modems that transferred 4kByte/s...



Over here

$ wget 
https://gitlab.com/lilypond/lilypond/-/releases/v2.24.0/downloads/lilypond-2.24.0-documentation.tar.xz

[...]
lilypond-2.24.0-documentation.t 
100%[=>] 166.25M  
3.42MB/s    in 2m 5s


[...] (1.33 MB/s) [...]

and my Internet connection is not what would be considered high-speed today.

A size increase is not to be happy about, but for documentation tarballs,
which most users who want them will only download once for every stable
release, it's not the end of the world.



OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Werner LEMBERG]

>> Indeed, but the manuals as a whole, in all languages, get also
>> distributed, and there it *does* make a significant difference
>> IMHO: Right now, the PDFs in `lilypond-2.24.0-documentation.tar.xz`
>> (which has a size of 170MByte) need 144MByte in total
>> (uncompressed).  Multiply the latter by four...
> 
> To me, it matters even less in the doc tarball because that is
> something you will only download once per release.

This is apparently a generation thing: I still can remember using
modems that transferred 4kByte/s...

BTW, I've open an issue regarding PDF font embedding/subsetting in
Cairo.

  https://gitlab.freedesktop.org/cairo/cairo/-/issues/618


 Werner

Re: Missing items to make Cairo ready

[Jean Abou Samra]

> Le 5 janv. 2023 à 14:24, Werner LEMBERG  a écrit :
> 
> Indeed, but the manuals as a whole, in all languages, get also
> distributed, and there it *does* make a significant difference IMHO:
> Right now, the PDFs in `lilypond-2.24.0-documentation.tar.xz` (which
> has a size of 170MByte) need 144MByte in total (uncompressed).
> Multiply the latter by four...


To me, it matters even less in the doc tarball because that is something you 
will only download once per release.


>> also wonder how people actually use the PDF manual. It's very big to
>> print out, and compared to the HTML manual, unwieldy to use on
>> screen.
> 
> In the HTML manual it is not possible to zoom into the images.  While
> the resolution of the PNG images is OK in general, there are sometimes
> situations where a non-pixel graphics format would be very beneficial.


Here is something we will be able to improve eventually thanks to Cairo, by 
embedding SVGs instead of PNGs. The current SVG backend is not good enough for 
that (slow, doesn’t embed fonts).

Re: Missing items to make Cairo ready

[Werner LEMBERG]

>> The only thing I would like to convince the Cairo people is to add
>> a mode to produce PDFs with font references instead of embedding –
>> and subsetting – fonts.  My Cairo knowledge is zero; maybe this is
>> already possible?
> 
> This makes no sense to me as a Cairo feature. Such PDF snippets
> would need custom post processing to assemble into the full
> document.

The general problem (i.e., the inclusion of multiple PDF images into a
larger PDF document) is not unique to LilyPond.  My idea is that there
exist special tools (for example, Ghostscript's 'pdfwrite' device or
`pdfsizeopt`) to produce size-optimized PDFs.[*] However, these tools
can only do their optimization job if the necessary preliminaries are
fulfilled.  If the PDF images contain subsetted fonts, it doesn't work
most of the time: You either need PDFs with complete fonts (i.e., not
subsetted), or PDFs with references to fonts.  I've now looked up the
Cairo manual, and it doesn't offer any control for that.

> If we do custom post processing, we might as well postprocess the
> final PDF directly to make all snippets point to a single music font
> object?

Font subsetting effectively prevents such post-processing since too
much information gets stripped off during this process.  For the
special case of Emmentaler fonts it might work because LilyPond knows
more about these fonts than for others, but not in the general case.

> IMO, working with a 35mb user manual isn't materially different from
> working with a 10mb user manual.  Both take a while to download.

Indeed, but the manuals as a whole, in all languages, get also
distributed, and there it *does* make a significant difference IMHO:
Right now, the PDFs in `lilypond-2.24.0-documentation.tar.xz` (which
has a size of 170MByte) need 144MByte in total (uncompressed).
Multiply the latter by four...

> also wonder how people actually use the PDF manual. It's very big to
> print out, and compared to the HTML manual, unwieldy to use on
> screen.

In the HTML manual it is not possible to zoom into the images.  While
the resolution of the PNG images is OK in general, there are sometimes
situations where a non-pixel graphics format would be very beneficial.


Werner


[*] With `pdfsizeopt` you can reduce the current size of the current
`notation.pdf` from 7.3MByte to 5MByte.

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 05/01/2023 à 13:23, Karlin High a écrit :

On Thu, Jan 5, 2023, 6:07 AM Han-Wen Nienhuys  wrote:

If we do custom post processing, we might as well postprocess the
final PDF directly to make all snippets point to a single music
font object?


I would like to hear more about what that could look like. There were 
numerous mentions of Poppler; can it do things like that?



No, it can't. It does not provide a way to change the document fonts 
before they are rendered.





OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Karlin High]

On Thu, Jan 5, 2023, 6:07 AM Han-Wen Nienhuys  wrote:

> If we do custom post processing, we might as well postprocess the final
> PDF directly to make all snippets point to a single music font object?
>

I would like to hear more about what that could look like. There were
numerous mentions of Poppler; can it do things like that?

This got me thinking of the "Engrave (Publish)" command in Frescobaldi.
Optimize output instead of processing only after the source is final.
--
Karlin High
Missouri, USA

>

Re: Missing items to make Cairo ready

[Han-Wen Nienhuys]

On Thu, Jan 5, 2023 at 7:16 AM Werner LEMBERG  wrote:

> The only thing I would like to convince the Cairo people is to add a
> mode to produce PDFs with font references instead of embedding – and
> subsetting – fonts.  My Cairo knowledge is zero; maybe this is already
> possible?

This makes no sense to me as a Cairo feature. Such PDF snippets would
need custom post processing to assemble into the full document. If we
do custom post processing, we might as well postprocess the final PDF
directly to make all snippets point to a single music font object?

IMO, working with a 35mb user manual isn't materially different from
working with a 10mb user manual. Both take a while to download. I also
wonder how people actually use the PDF manual. It's very big to print
out, and compared to the HTML manual, unwieldy to use on screen.

-- 
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 05/01/2023 à 07:16, Werner LEMBERG a écrit :

Well, 'dropping the old PS backend' is something we should do very
carefully.  Hopefully, we are first marking the old backend as
deprecated in the next major release before removing it completely in
the release after next.




Yes, but the time to figure out what need to be ready by then is *now*.
It will take a lot of time to get each of the features we need in Cairo
written, reviewed, merged, released and to wait for that release to be
reasonably current for Linux distros to pick it up.

I strongly disagree that PDFs of the same size as today is a requirement
for this switch as long as the size stays reasonable, and unlike 
output-attributes,

I am not willing to put time into trying to contribute font references
to Cairo.



The only thing I would like to convince the Cairo people is to add a
mode to produce PDFs with font references instead of embedding – and
subsetting – fonts.  My Cairo knowledge is zero; maybe this is already
possible?



Nope.



OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Werner LEMBERG]

>> We could then get rid of Ghostscript completely.

Mhmm, please ignore this sentence, it doesn't make sense.

> So, to be clear, in your opinion, solving this is a requirement
> before we drop the old PS backend?

Well, 'dropping the old PS backend' is something we should do very
carefully.  Hopefully, we are first marking the old backend as
deprecated in the next major release before removing it completely in
the release after next.

The only thing I would like to convince the Cairo people is to add a
mode to produce PDFs with font references instead of embedding – and
subsetting – fonts.  My Cairo knowledge is zero; maybe this is already
possible?


Werner

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 04/01/2023 à 23:41, David Kastrup a écrit :

Again, the size of a manual that will get copied repeatedly and
indefinitely and the size of a document that is transferred once to a
printer are different things, and the quoted reply of a Ghostscript
developer suggests that they are not willing or able to view the
realities of one use case as similarly important or relevant as the
other.




We are going around in circles. However hard we work, there will be
a tradeoff at the point we drop the PS backend. I definitely don't
think the size increase of notation.pdf from 5MB to 20MB outweighs
the cost of maintaining the PS backend.

Cairo's primary focus is not on PDF features. If I were a Cairo
maintainer vetting a patch to add an option to disable font embedding
because "we have this niche use case for it, and the developers
of the main tool to work with PDFs don't want to support our
use case, so we need Cairo to work around it", I would say no.



OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[David Kastrup]

Jean Abou Samra  writes:

> Le 04/01/2023 à 22:42, David Kastrup a écrit :
>> There is a difference between a score that is downloaded and copied and
>> transferred thousands of times indefinitely, and a finalized print
>> journal that is sent from a publisher to a printer once.
>
>
>
> I think there is a misunderstanding. We are talking about the
> LilyPond manuals only, not about the size of user scores.

Again, the size of a manual that will get copied repeatedly and
indefinitely and the size of a document that is transferred once to a
printer are different things, and the quoted reply of a Ghostscript
developer suggests that they are not willing or able to view the
realities of one use case as similarly important or relevant as the
other.

-- 
David Kastrup

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 04/01/2023 à 22:49, Werner LEMBERG a écrit :

Well, in this case it's the Cairo people we would have to convince.
We could then get rid of Ghostscript completely.




So, to be clear, in your opinion, solving this is a requirement
before we drop the old PS backend? Waiting for a potentially
complex addition that the Cairo people may or may not accept,
if someone even writes it?

Quite frankly, I think it's not going to happen until the 12th
of Never.




OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Werner LEMBERG]

> And again, it's not like the NR PDF was unusable when it was
> ~35 MB in LilyPond 2.18.
> 
> What alternative would you propose?

In the next days I will investigate how to get small PDFs with the new
PDF engine of Ghostscript, reporting problems to the maintainers.
Let's see what this will bring.

> (FYI, your mail client is still repeating the subject.)

Sorry for that.  I haven't found the time yet to analyze and report it
to the Mew mail client people.  Interestingly, it's only affecting
e-mails coming from you, Jean...


Werner

Re: Missing items to make Cairo ready

[Werner LEMBERG]

>> Honestly, a fourfold size increase is not something that I would
>> call 'acceptable'.
> 
> Upstream might need some convincing.
> 
> Ghostscript's Ken Sharp in 2017:
> 
> "When we have customers wanting to send us 125GB files I have to say
> that a concern over file sizes in the few megabytes seems a bit
> picky."
> 
> <https://lists.gnu.org/archive/html/lilypond-devel/2017-09/msg00131.html>

Well, in this case it's the Cairo people we would have to convince.
We could then get rid of Ghostscript completely.


Werner

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 04/01/2023 à 22:42, David Kastrup a écrit :

There is a difference between a score that is downloaded and copied and
transferred thousands of times indefinitely, and a finalized print
journal that is sent from a publisher to a printer once.




I think there is a misunderstanding. We are talking about the
LilyPond manuals only, not about the size of user scores.



OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[David Kastrup]

Jean Abou Samra  writes:

> Le 04/01/2023 à 22:22, Werner LEMBERG a écrit :
>> Honestly, a fourfold size increase is not something that I would call
>> 'acceptable'.
>
>
>
> It's a question of absolute values, not relative ones. A fourfold
> increase from an acceptable value to an acceptable value is
> acceptable.

There is a difference between a score that is downloaded and copied and
transferred thousands of times indefinitely, and a finalized print
journal that is sent from a publisher to a printer once.

-- 
David Kastrup

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 04/01/2023 à 22:22, Werner LEMBERG a écrit :

Honestly, a fourfold size increase is not something that I would call
'acceptable'.




It's a question of absolute values, not relative ones. A fourfold
increase from an acceptable value to an acceptable value is
acceptable.

Due to browser caching, you will not download the manual that
often, only if you use a new machine, or your browser cache
filled too much and you had not consulted it in a while.
If you consult the documentation locally, you will download
it only once.

And again, it's not like the NR PDF was unusable when it was
~35 MB in LilyPond 2.18.

What alternative would you propose?

(FYI, your mail client is still repeating the subject.)



OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Karlin High]

On 1/4/2023 3:22 PM, Werner LEMBERG wrote:

Honestly, a fourfold size increase is not something that I would call
'acceptable'.


Upstream might need some convincing.

Ghostscript's Ken Sharp in 2017:

"When we have customers wanting to send us 125GB files I have to say 
that a concern over file sizes in the few megabytes seems a bit picky."



--
Karlin High
Missouri, USA

Re: Missing items to make Cairo ready,Re: Missing items to make Cairo ready,Re: Missing items to make Cairo ready,Re: Missing items to make Cairo ready

[Werner LEMBERG]

>> The idea is that you compile all snippets with references to fonts
>> only but not with actual fonts.  This allows Ghostscript to analyze
>> and squeeze the fonts of all included PDFs globally; as an example,
>> the NR then has only a few dozen (subsetted) fonts instead of about
>> 4500.
> 
> OK, this sounds like it will be tough to do in the Cairo
> backend.

Actually, it would be *great* if such a feature could be incorporated
into Cairo itself.  I guess LilyPond is not the only application of
this library that would love to include other PDFs, creating
small-sized output files.

> Given that the size still looks acceptable (I mean, with browser
> caching, you don't actually download doc PDFs often, and it's still
> more than twice smaller than before extractpfmark was introduced in
> 2016), I think we can just let these options go, and possibly remove
> extractpdfmark from our doc build eventually if the size gains are
> no longer worth it.

Honestly, a fourfold size increase is not something that I would call
'acceptable'.


Werner

Cairo label

[Jean Abou Samra]

Tangential to the ongoing Cairo discussions: does anybody
mind adding a ~Cairo label for Cairo-related issues?



OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready, Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 04/01/2023 à 16:13, Werner LEMBERG a écrit :

The idea is that you compile all snippets with references to fonts
only but not with actual fonts.  This allows Ghostscript to analyze
and squeeze the fonts of all included PDFs globally; as an example,
the NR then has only a few dozen (subsetted) fonts instead of about
4500.



OK, this sounds like it will be tough to do in the Cairo backend. Given
that the size still looks acceptable (I mean, with browser caching,
you don't actually download doc PDFs often, and it's still more than
twice smaller than before extractpfmark was introduced in 2016),
I think we can just let these options go, and possibly remove extractpdfmark
from our doc build eventually if the size gains are no longer worth it.



OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready,Re: Missing items to make Cairo ready

[Werner LEMBERG]

> With extractpdfmark
> ===
> 
> PS backend:
> 
> 4,8M    Documentation/out-www/en/notation.pdf

FYI: Using a recent GS version with its new PDF backend, the size gets
larger than 20MByte.  I guess this is a bug, and I will soon report
this to the GS people so that it gets fixed in the next GS version (in
March), which will eventually drop the old PDF engine completely.

On the other hand, GS's new PDF engine no longer contains a PS
interpreter; it is possible that `extractpdfmark` doesn't work
anymore, and we have to find something new...

> I'm not sure what exactly yields the smaller size of PS backend +
> extractpdfmark or how we could get it in Cairo.

The idea is that you compile all snippets with references to fonts
only but not with actual fonts.  This allows Ghostscript to analyze
and squeeze the fonts of all included PDFs globally; as an example,
the NR then has only a few dozen (subsetted) fonts instead of about
4500.

`extractpdfmark` works around another 'feature' of GS.[*]  From
extractpdfmark's `README.md` file:

  [...] when Ghostscript reads the main PDF produced by the TeX system
  and outputs the final PDF it does not preserve PDF page-mode and
  named-destinations etc.  As a result, when you open the final PDF,
  it is not displayed correctly.  Also, remote PDF links will not work
  correctly.  [...]

  This program is able to extract page mode and named destinations as
  PDFmark from PDF.  By using this you can get the small PDF files
  that have preserved them.


Werner


[*] Probably driven by customer demands, the GS people removed a lot
of PDF manipulation possibilities over the last years,
concentrating rather on creating PDFs that are good for printing.
Using a new PDF engine is another step into this direction; this
time, controlling Ghostscript via PostScript features gets lost.

Re: Missing items to make Cairo ready

[Jonas Hahnfeld via Discussions on LilyPond development]

On Wed, 2023-01-04 at 15:00 +0100, Jean Abou Samra wrote:
> I was initially scared by Poppler's dependencies, but now that I look at 
> its CMakeLists.txt file, it seems that most of them are optional, which 
> makes it more palatable. I'm still not sure what you lose if you don't 
> compile with GTK, though, and I can't say I really understand this stuff 
> well anyway.

As far as I understand the CMakeLists.txt, it's only used for gtk-doc
which is off by default anyway and certainly not a runtime dependency.
On the other hand, cross-compilation with CMake is ... not exactly the
nicest stroll in the garden (and a first in release/binaries).

On the other hand, librsvg is written in Rust where I have no
experience how practical it actually is to integrate into our binaries
on all platforms.


signature.asc
Description: This is a digitally signed message part

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 04/01/2023 à 14:32, Jonas Hahnfeld a écrit :

Right: as far as I understand, these are also crucial for our own
documentation build to end up with reasonably sized PDFs. Did you check
what the situation is if we used snippets compiled with Cairo?



Without extractpdfmark
==


PS backend:

42M    Documentation/out-www/en/notation.pdf


Cairo backend:

16M    ./Documentation/out-www/en/notation.tmp.pdf


With extractpdfmark
===

PS backend:

4,8M    Documentation/out-www/en/notation.pdf


Cairo backend:

14M    Documentation/out-www/en/notation.pdf



Not that bad?

I'm not sure what exactly yields the smaller size of
PS backend + extractpdfmark or how we could get it
in Cairo.



OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 04/01/2023 à 14:29, Jonas Hahnfeld a écrit :

On Wed, 2023-01-04 at 13:39 +0100, Jean Abou Samra wrote:

Le 04/01/2023 à 12:52, Han-Wen Nienhuys a écrit :

PNG images will always be clunky for embedding line art, so it can't
be the recommended solution. What makes most sense for users? Other
illustration programs also can't process EPS (for the same reasons we
don't), so how do those programs embed line art? What is the preferred
format for logos today? SVG? AI? PDF?

PNG?

... and PDF, if the logo is created by a paid designer (probably
exported from Adobe Illustrator) and it's "important" for print, for
example large posters. At least looking at my previous university and
my current employer (actually both also provide(d) EPS!).



I was initially scared by Poppler's dependencies, but now that I look at 
its CMakeLists.txt file, it seems that most of them are optional, which 
makes it more palatable. I'm still not sure what you lose if you don't 
compile with GTK, though, and I can't say I really understand this stuff 
well anyway.




OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Jonas Hahnfeld via Discussions on LilyPond development]

On Wed, 2023-01-04 at 14:00 +0100, Jean Abou Samra wrote:
> Le 29/12/2022 à 01:53, Jean Abou Samra a écrit :
> > Hi,
> > 
> > I have just opened issues for the missing features of
> > the Cairo backend that I am aware of.
> > 
> > https://gitlab.com/lilypond/lilypond/-/issues/6500
> > https://gitlab.com/lilypond/lilypond/-/issues/6501
> > https://gitlab.com/lilypond/lilypond/-/issues/6502
> > https://gitlab.com/lilypond/lilypond/-/issues/6503
> > https://gitlab.com/lilypond/lilypond/-/issues/6504
> > 
> > Are there any others?
> 
> The topic of this thread has deviated somewhat from the original
> question, and I don't really mind, but I would still like to tackle
> the original question.
> 
> Here are a few things that can't be supported in Cairo:
> 
> - -dfont-export-dir=...
> - -dfont-ps-resdir=...
> - -dgs-api (but makes no sense in Cairo)
> - -dgs-load-fonts
> - -dgs-load-lily-fonts
> - -dgs-never-embed-fonts
> - -dno-include-eps-fonts
> 
> AFAICS, these only make sense if you're using GS to convert
> the PS output to PDF yourself manually, so they're obviated
> in Cairo, but I would like to check that this is the case.

Right: as far as I understand, these are also crucial for our own
documentation build to end up with reasonably sized PDFs. Did you check
what the situation is if we used snippets compiled with Cairo?

Jonas


signature.asc
Description: This is a digitally signed message part

Re: Missing items to make Cairo ready

[Jonas Hahnfeld via Discussions on LilyPond development]

On Wed, 2023-01-04 at 13:39 +0100, Jean Abou Samra wrote:
> Le 04/01/2023 à 12:52, Han-Wen Nienhuys a écrit :
> > PNG images will always be clunky for embedding line art, so it can't
> > be the recommended solution. What makes most sense for users? Other
> > illustration programs also can't process EPS (for the same reasons we
> > don't), so how do those programs embed line art? What is the preferred
> > format for logos today? SVG? AI? PDF?
> 
> PNG?

... and PDF, if the logo is created by a paid designer (probably
exported from Adobe Illustrator) and it's "important" for print, for
example large posters. At least looking at my previous university and
my current employer (actually both also provide(d) EPS!).


signature.asc
Description: This is a digitally signed message part

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 29/12/2022 à 01:53, Jean Abou Samra a écrit :

Hi,

I have just opened issues for the missing features of
the Cairo backend that I am aware of.

https://gitlab.com/lilypond/lilypond/-/issues/6500
https://gitlab.com/lilypond/lilypond/-/issues/6501
https://gitlab.com/lilypond/lilypond/-/issues/6502
https://gitlab.com/lilypond/lilypond/-/issues/6503
https://gitlab.com/lilypond/lilypond/-/issues/6504

Are there any others?




The topic of this thread has deviated somewhat from the original
question, and I don't really mind, but I would still like to tackle
the original question.

Here are a few things that can't be supported in Cairo:

- -dfont-export-dir=...
- -dfont-ps-resdir=...
- -dgs-api (but makes no sense in Cairo)
- -dgs-load-fonts
- -dgs-load-lily-fonts
- -dgs-never-embed-fonts
- -dno-include-eps-fonts


AFAICS, these only make sense if you're using GS to convert
the PS output to PDF yourself manually, so they're obviated
in Cairo, but I would like to check that this is the case.

There is also -dstrokeadjust. Not sure if we need a Cairo
equivalent.

Also -dsvg-woff.

Jean



OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 04/01/2023 à 12:52, Han-Wen Nienhuys a écrit :

On Thu, Dec 29, 2022 at 1:53 AM Jean Abou Samra  wrote:

Hi,

I have just opened issues for the missing features of
the Cairo backend that I am aware of.

https://gitlab.com/lilypond/lilypond/-/issues/6500
https://gitlab.com/lilypond/lilypond/-/issues/6501
https://gitlab.com/lilypond/lilypond/-/issues/6502
https://gitlab.com/lilypond/lilypond/-/issues/6503
https://gitlab.com/lilypond/lilypond/-/issues/6504

Are there any others?

I've read through the discussion you started here, but IMO we're too
fixated on backward compatibility, which is unfortunate because the PS
-> GS -> PDF route was never thought through carefully. We started
dumping PostScript because we were going through TeX before, and
creating PDF at the time involved .tex => .dvi => .ps  => .pdf (with
.eps included into .dvi for graphics).  However, at the time, PDF
rather than PS was already the common format for print documents. We
were also lazy, and didn't want to bother learning how to dump PDF
directly.

PNG images will always be clunky for embedding line art, so it can't
be the recommended solution. What makes most sense for users? Other
illustration programs also can't process EPS (for the same reasons we
don't), so how do those programs embed line art? What is the preferred
format for logos today? SVG? AI? PDF?




PNG?

Look here:

https://www.gnu.org/graphics/package-logos.fr.html

More than half of the logos are in PNG, some are in SVG, a
some in JPG.

I still don't understand the criticism against logos in PNG. For
something like a page watermark, I understand that vector graphics
may be important, but for something the size of a logo?

I looked a little bit at librsvg. On the plus side, their
COMPILING.md describes how to cross-compile it with MinGW,
so we know that it works. On the minus side, it pulls in
Gdk-pixbuf and Libxml2.




Given the sorry state of the SVG backend, and the niche quality of the
output-attributes feature, we could be justified to drop the SVG
backend rather than implementing attributes in Cairo-SVG (although it
sounds like a straightforward extension to Cairo).



I don't think so; look at the kind of stuff it would break:

https://github.com/paopre/Spontini



OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Han-Wen Nienhuys]

On Thu, Dec 29, 2022 at 1:53 AM Jean Abou Samra  wrote:
>
> Hi,
>
> I have just opened issues for the missing features of
> the Cairo backend that I am aware of.
>
> https://gitlab.com/lilypond/lilypond/-/issues/6500
> https://gitlab.com/lilypond/lilypond/-/issues/6501
> https://gitlab.com/lilypond/lilypond/-/issues/6502
> https://gitlab.com/lilypond/lilypond/-/issues/6503
> https://gitlab.com/lilypond/lilypond/-/issues/6504
>
> Are there any others?

I've read through the discussion you started here, but IMO we're too
fixated on backward compatibility, which is unfortunate because the PS
-> GS -> PDF route was never thought through carefully. We started
dumping PostScript because we were going through TeX before, and
creating PDF at the time involved .tex => .dvi => .ps  => .pdf (with
.eps included into .dvi for graphics).  However, at the time, PDF
rather than PS was already the common format for print documents. We
were also lazy, and didn't want to bother learning how to dump PDF
directly.

PNG images will always be clunky for embedding line art, so it can't
be the recommended solution. What makes most sense for users? Other
illustration programs also can't process EPS (for the same reasons we
don't), so how do those programs embed line art? What is the preferred
format for logos today? SVG? AI? PDF?

Given the sorry state of the SVG backend, and the niche quality of the
output-attributes feature, we could be justified to drop the SVG
backend rather than implementing attributes in Cairo-SVG (although it
sounds like a straightforward extension to Cairo).

Regarding versioning: the 1.x to 2.x transition was motivated by
radical syntax changes that necessitated converting and 'manually'
verifying the .ly files. Since Cairo vs. Ghostscript doesn't affect
the semantics of .ly files, I think we can continue the 2.x version
number. As a practical example, page layout was introduced in 2.4, and
direct to PostScript only became default in 2.6; both changes are much
more invasive than what we are discussing here.


> I ask this because we are at an early point in the
> 2.26 release cycle, which could potentially be ideal
> to get testing for "Cairo by default" if it were
> ready, but it isn't yet.
>
> Best,
> Jean
>


--
Han-Wen Nienhuys - hanw...@gmail.com - http://www.xs4all.nl/~hanwen

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 02/01/2023 à 01:30, Luca Fascione a écrit :


On Sun, Jan 1, 2023 at 11:16 PM Jean Abou Samra  
wrote:


 Its image operator doesn't have
support for transparency on some pixels and not others.


So you're worried about PS with embedded pixels coming from maybe a 
PNG not roundtripping correctly back to PNG when transparency is involved?
And you're saying the reason users would do it this way is because 
there is no support in lilypond for embedded pixel rasters except via EPS?



I don't understand all of what you are writing, but it doesn't look like 
we're on the same page.


In !1787, I am adding a command \image, which can embed PNG files in 
addition to EPS files. Unlike the state of the MR when this thread 
began, this command now works in all ouput backends: PS, SVG, Cairo.


In the Cairo backend, the implementation lets Cairo read the pixels from 
the PNG file into a memory buffer, and write that as image onto the 
output surface.


In the SVG backend, the implementation embeds the image using an  
tag containing the image data as base64.


In the PS backend, it uses the PostScript 'image' operator with embedded 
image data in hexadecimal (Cairo is actually used internally here to 
parse the PNG file and write it as a stream that PostScript understands).


The problem with the PS backend implementation is that PostScript 
doesn't have a way to embed images with alpha transparency.


Note that the Cairo backend *can* write PostScript files with 
alpha-transparent PNGs, but that is because it understands the PNG image 
and all of the other drawings. It transforms the alpha-transparent PNG 
into a PNG without alpha transparency, by combining it with the other 
elements on the page. LilyPond is not graphics software and cannot do 
that. The PS backend delegates to Ghostscript as graphics software, and 
Ghostscript does not have this ability.


It's not "users" who do the PNG → PS conversion, but LilyPond, internally.




OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Luca Fascione]

On Sun, Jan 1, 2023 at 11:16 PM Jean Abou Samra  wrote:

>  Its image operator doesn't have
> support for transparency on some pixels and not others.
>

So you're worried about PS with embedded pixels coming from maybe a PNG not
roundtripping correctly back to PNG when transparency is involved?
And you're saying the reason users would do it this way is because there is
no support in lilypond for embedded pixel rasters except via EPS?

If this is all you're worried about, why not just support a command to
inject a raster image and a separate one supporting postscript images
and have the latter support transparency when coming from vector graphics
only? (I seem to recall this was also suggested, but I did not
appreciate the meaning of the difference at the time)

The use case that was discussed earlier in this thread, where a user might
want to embed a publisher's logo in a pdf would
appear to be most relevant when the file to be embedded is vector graphics,
and I do agree with who it was who brought it up
that it would be disappointing to find it auto-rasterized magically (was it
Harm maybe?). Also I've been asking myself what reference
resolution would that use? How will a user control it?

L

-- 
Luca Fascione

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 01/01/2023 à 12:36, Jean Abou Samra a écrit :

Le 30/12/2022 à 13:08, Jean Abou Samra a écrit :

which means figuring out how to do PNGs via the default PS
backend and GS.



I looked a bit at this.

It's not insurmountable, *but*, alpha transparency is not going
to work. PNG images with an alpha channel will need to be
converted to plain RGB because PostScript doesn't support
alpha transparency, and while GhostScript does have a little
support for it, it seems the only way to have an image with
variable levels of transparency across the image is to draw
each of the pixels separately.

Does that sound acceptable?

(For completeness: PNG images in the default SVG backend should be
easy.)



Added now, with the caveat above.

As said on the MR, an option would be to disable transparency by default 
so that the default config renders the same on all backends, and make 
the config with transparency enabled warn if the PS backend is used.




OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 01/01/2023 à 23:10, Luca Fascione a écrit :
Sorry I'm confused: my imagemagick point was that it seems to 
support EPS -> PNG with alpha, see for example


https://legacy.imagemagick.org/discourse-server/viewtopic.php?t=15985

(it seems that if the input EPS is CMYK, one needs to convert to RGB 
first, and then alpha flows through fine, I have not had the time to 
double check this).


My thought was that however it is that ImageMagick drives GS to 
achieve this, you could do as well?
I have arguably been casual as to whether the PostScript file is 
encapsulated or not, assuming it would make no difference for this aspect.



The problem isn't converting EPS to PNG with alpha, but the other way 
around.


PS/EPS code read by Ghostscript can contain transparency operators, but 
Ghostscript only supports operators that set the transparency, e.g., for 
the current path, like the color. Its image operator doesn't have 
support for transparency on some pixels and not others.





OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Luca Fascione]

Sorry I'm confused: my imagemagick point was that it seems to support EPS
-> PNG with alpha, see for example

https://legacy.imagemagick.org/discourse-server/viewtopic.php?t=15985

(it seems that if the input EPS is CMYK, one needs to convert to RGB first,
and then alpha flows through fine, I have not had the time to double check
this).

My thought was that however it is that ImageMagick drives GS to achieve
this, you could do as well?
I have arguably been casual as to whether the PostScript file is
encapsulated or not, assuming it would make no difference for this aspect.

L

On Sun, Jan 1, 2023 at 8:19 PM Jean Abou Samra  wrote:

> Le 01/01/2023 à 19:30, Luca Fascione a écrit :
> > Also, how does ImageMagick implement PS to PNG conversion?
>
> Through Ghostscript, just like LilyPond does now.
>



-- 
Luca Fascione

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 01/01/2023 à 19:30, Luca Fascione a écrit :
What if you use GS to do PS to PDF instead and embed that? Would Cairo 
let you?

(For the PDF backend I mean)



Cf. the remark above about Poppler.

(It would work for all output formats.)



Also, how does ImageMagick implement PS to PNG conversion?



Through Ghostscript, just like LilyPond does now.



OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Luca Fascione]

What if you use GS to do PS to PDF instead and embed that? Would Cairo let
you?
(For the PDF backend I mean)

Also, how does ImageMagick implement PS to PNG conversion? I wonder if
their approach might inspire us, or whether having an optional dependency
would be an idea to consider (if you have imagemagick and/or use some
option to Lily you get one behaviour vs the other).

L


On Sun, 1 Jan 2023, 12:36 Jean Abou Samra,  wrote:

> Le 30/12/2022 à 13:08, Jean Abou Samra a écrit :
> > which means figuring out how to do PNGs via the default PS
> > backend and GS.
>
>
> I looked a bit at this.
>
> It's not insurmountable, *but*, alpha transparency is not going
> to work. PNG images with an alpha channel will need to be
> converted to plain RGB because PostScript doesn't support
> alpha transparency, and while GhostScript does have a little
> support for it, it seems the only way to have an image with
> variable levels of transparency across the image is to draw
> each of the pixels separately.
>
> Does that sound acceptable?
>
> (For completeness: PNG images in the default SVG backend should be
> easy.)
>

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 30/12/2022 à 13:08, Jean Abou Samra a écrit :

which means figuring out how to do PNGs via the default PS
backend and GS.



I looked a bit at this.

It's not insurmountable, *but*, alpha transparency is not going
to work. PNG images with an alpha channel will need to be
converted to plain RGB because PostScript doesn't support
alpha transparency, and while GhostScript does have a little
support for it, it seems the only way to have an image with
variable levels of transparency across the image is to draw
each of the pixels separately.

Does that sound acceptable?

(For completeness: PNG images in the default SVG backend should be
easy.)


OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 01/01/2023 à 00:40, Paolo Prete a écrit :



On Sat, Dec 31, 2022 at 1:39 AM Jean Abou Samra  
wrote:


Le 31/12/2022 à 01:29, Paolo Prete a écrit :
> Yeah, I also thought about overlying multiple surfaces.

Nope, no dice.

> But don't exclude my tip: after all, the colors can give you unique
> identifiers for the map I've described to you, and this makes the
> mechanism safe


Well, it's pretty hacky.


I know but I would not underestimate it. I found the following merge 
request, where I see your participation:


https://gitlab.freedesktop.org/cairo/cairo/-/merge_requests/254

In theory, adding  groups with output-attributes should have a 
similar, if not identical implementation.



I think it should be simpler because it does not need to get the extents 
of the inner content, unlike hyperlinks.



What I fear is that manipulating the SVG tree is beyond the scope of 
the library, which is meant for _rendering_; but at the same time I 
think that producing a SVG file is in itself more than "rendering" 
(then, these features could be accepted).



Uli Schlachter is a Cairo maintainer, and her/his (?) reaction on the MR 
was merely commenting on the code, but not making objections on the 
principle.


A few years ago, the PDF backend gained lots of non-graphical features 
(which LilyPond is now making use of: metadata, soon outlines, etc.)


https://gitlab.freedesktop.org/cairo/cairo/-/issues/281 has been open 
for 12 years and nobody turned it down.


I think it is in the scope of the library. It just needs an implementation.


Therefore, I think that the best thing is to ping the cairo staff and 
ask if both these features have chances to be accepted, considering 
the enormous advantage they have for LilyPond and other applications 
as well.



As you wish. I predict they will respond "sure, but are you going to 
submit the code"?





OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Paolo Prete]

On Sat, Dec 31, 2022 at 1:39 AM Jean Abou Samra  wrote:

> Le 31/12/2022 à 01:29, Paolo Prete a écrit :
> > Yeah, I also thought about overlying multiple surfaces.
>
> Nope, no dice.
>
> > But don't exclude my tip: after all, the colors can give you unique
> > identifiers for the map I've described to you, and this makes the
> > mechanism safe
>
>
> Well, it's pretty hacky.
>
>
I know but I would not underestimate it. I found the following merge
request, where I see your participation:

https://gitlab.freedesktop.org/cairo/cairo/-/merge_requests/254

In theory, adding  groups with output-attributes should have a similar,
if not identical implementation.
What I fear is that manipulating the SVG tree is beyond the scope of the
library, which is meant for _rendering_; but at the same time I think that
producing a SVG file is in itself more than "rendering" (then, these
features could be accepted).
Therefore, I think that the best thing is to ping the cairo staff and ask
if both these features have chances to be accepted, considering the
enormous advantage they have for LilyPond and other applications as well.


>
>

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 31/12/2022 à 01:39, Jean Abou Samra a écrit :

I think we'd better figure out how to patch Cairo to support this.


(To clarify: I mean "write a patch to submit upstream", not "apply a 
patch in the release infrastructure").




OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 31/12/2022 à 01:29, Paolo Prete a écrit :

Yeah, I also thought about overlying multiple surfaces.


Nope, no dice.

But don't exclude my tip: after all, the colors can give you unique 
identifiers for the map I've described to you, and this makes the 
mechanism safe



Well, it's pretty hacky.

I think we'd better figure out how to patch Cairo to support this. The 
source of the SVG backend is here:


https://gitlab.freedesktop.org/cairo/cairo/-/blob/master/src/cairo-svg-surface.c

I don't think it's hard to add a function for inserting output 
attributes to this. It just takes time to figure out the design from the 
4500 lines of C.





OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Paolo Prete]

Yeah, I also thought about overlying multiple surfaces.
But don't exclude my tip: after all, the colors can give you unique
identifiers for the map I've described to you, and this makes the mechanism
safe

On Sat, Dec 31, 2022 at 1:23 AM Jean Abou Samra  wrote:

> Le 31/12/2022 à 01:13, Paolo Prete a écrit :
> > I'll examine it, thanks.
> > Meanwhile, I don't know if it is the right case, but it comes to my
> > mind a hack, which I applied on some similar code:
> >
> > Before the SVG bytestream is emitted by the write_func, I would set a
> > temporary, special and different _color_ for each grob that is
> > associated with output-attributes.
> > This color is automatically generated and must obviously be different
> > from any other color that will be included in the output SVG.
> > Then you can create a map with key = specialcolor, value = pair
> > (original color,  output-attributes) and when you parse the generated
> > bytestream, if you find any set of objects with a color included in
> > the map, you can group them, set output attributes on the group and
> > restore the original color.
> >
> > I repeat: this is a hack, it's a bit imaginative and I don't know if
> > it's really suitable for the present case. But the unique color
> > mechanism is much safer than you might think.
>
>
> Wait, I'm starting to wonder if it's possible via a second surface.
>
> I'll try this, but not guarantees.
>
>

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 31/12/2022 à 01:13, Paolo Prete a écrit :

I'll examine it, thanks.
Meanwhile, I don't know if it is the right case, but it comes to my 
mind a hack, which I applied on some similar code:


Before the SVG bytestream is emitted by the write_func, I would set a 
temporary, special and different _color_ for each grob that is 
associated with output-attributes.
This color is automatically generated and must obviously be different 
from any other color that will be included in the output SVG.
Then you can create a map with key = specialcolor, value = pair 
(original color,  output-attributes) and when you parse the generated 
bytestream, if you find any set of objects with a color included in 
the map, you can group them, set output attributes on the group and 
restore the original color.


I repeat: this is a hack, it's a bit imaginative and I don't know if 
it's really suitable for the present case. But the unique color 
mechanism is much safer than you might think.



Wait, I'm starting to wonder if it's possible via a second surface.

I'll try this, but not guarantees.



OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Paolo Prete]

I'll examine it, thanks.
Meanwhile, I don't know if it is the right case, but it comes to my mind a
hack, which I applied on some similar code:

Before the SVG bytestream is emitted by the write_func, I would set a
temporary, special and different _color_ for each grob that is associated
with output-attributes.
This color is automatically generated and must obviously be different from
any other color that will be included in the output SVG.
Then you can create a map with key = specialcolor, value = pair (original
color,  output-attributes) and when you parse the generated bytestream, if
you find any set of objects with a color included in the map, you can group
them, set output attributes on the group and restore the original color.

I repeat: this is a hack, it's a bit imaginative and I don't know if it's
really suitable for the present case. But the unique color mechanism is
much safer than you might think.




On Sat, Dec 31, 2022 at 12:41 AM Jean Abou Samra  wrote:

> Le 31/12/2022 à 00:29, Paolo Prete a écrit :
> > Hi Jean,
> >
> > Do you have a test snippet for cairo_svg_create_for_stream() ?
> >
> > I would like to make some tests and I googled a bit but could not find
> > anything useful for it.
>
>
> Fortunately, my deleted branch for this was still around in my Git
> repository. I ressuscitated it and uploaded it to my fork, so you should
> be able to check it out with
>
> git fetch "g...@gitlab.com:jeanas/lilypond.git" cairo-output-attributes
> git checkout -b cairo-output-attributes FETCH_HEAD
>
> You can see that the  tags are properly output, but before all the
> rest of the SVG output.
>
>
>

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 31/12/2022 à 00:29, Paolo Prete a écrit :

Hi Jean,

Do you have a test snippet for cairo_svg_create_for_stream() ?

I would like to make some tests and I googled a bit but could not find 
anything useful for it.



Fortunately, my deleted branch for this was still around in my Git 
repository. I ressuscitated it and uploaded it to my fork, so you should 
be able to check it out with


git fetch "g...@gitlab.com:jeanas/lilypond.git" cairo-output-attributes
git checkout -b cairo-output-attributes FETCH_HEAD

You can see that the  tags are properly output, but before all the 
rest of the SVG output.





OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Paolo Prete]

Hi Jean,

Do you have a test snippet for cairo_svg_create_for_stream() ?

I would like to make some tests and I googled a bit but could not find
anything useful for it.

Thanks



Il ven 30 dic 2022, 18:17 Jean Abou Samra  ha scritto:

> Le 30/12/2022 à 18:12, Paolo Prete a écrit :
> >
> > Hello,
> >
> > I'm probably saying something unfeasible and a dirty mix; nevertheless
> > I wonder: could there be a way to use the cairo backend as a base,
> > along with SVG backend snippets limited to the grobs for which the
> > output-attributes property has been set?
> >
>
>
> https://gitlab.com/lilypond/lilypond/-/issues/6503#note_1223886041
>
>

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 30/12/2022 à 18:12, Paolo Prete a écrit :


Hello,

I'm probably saying something unfeasible and a dirty mix; nevertheless 
I wonder: could there be a way to use the cairo backend as a base, 
along with SVG backend snippets limited to the grobs for which the 
output-attributes property has been set?





https://gitlab.com/lilypond/lilypond/-/issues/6503#note_1223886041



OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Paolo Prete]

On Fri, Dec 30, 2022 at 1:55 PM Jonas Hahnfeld via Discussions on LilyPond
development  wrote:

>
> Ok, this is SVG which we can probably agree that the backend is in a
> bad state. At some point, there was a discussion on making Cairo the
> default for producing SVGs but IIRC it faded out with the lack of
> support for output-attributes, ie
> https://gitlab.com/lilypond/lilypond/-/issues/6503
>
> Once that is solved, only enabling Cairo by default for SVGs may be a
> separate discussion that is worth having.
>
>
Hello,

I'm probably saying something unfeasible and a dirty mix; nevertheless I
wonder: could there be a way to use the cairo backend as a base, along with
SVG backend snippets limited to the grobs for which the output-attributes
property has been set?

Re: Missing items to make Cairo ready

[Lukas-Fabian Moser]

A logo will occupy a small fraction of the page. Do you really need 
to have it in such high resolution that the file size becomes a 
problem?


In practice, probably. (Although there are also things like 
page-sized watermarks.)

Given "but" below, did you miss a "not" in this sentence?

Yes, sorry.

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 30/12/2022 à 17:39, Lukas-Fabian Moser a écrit :


Am 30.12.22 um 17:34 schrieb Jean Abou Samra:

Le 30/12/2022 à 17:20, Lukas-Fabian Moser a écrit :
I agree with Jonas that it would be a considerable loss not to be 
able to import vector graphics. I don't concur with Jean's argument 
that one can replace them by using markup commands: One obvious use 
case for proper vector import would be including a publisher's logo, 
for instance.
A logo will occupy a small fraction of the page. Do you really need 
to have it in such high resolution that the file size becomes a problem?


In practice, probably. (Although there are also things like page-sized 
watermarks.)



Given "but" below, did you miss a "not" in this sentence?



OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 30/12/2022 à 17:41, David Kastrup a écrit :

So?  ps2pdf exists.  It's not like it isn't already part of our toolkit,
so I don't see the problem with using it for creating a vector format.
pdftocairo can create an SVG from that, for example.



Read the question again. It was about outputting LilyPond markup 
commands from the EPS code.




OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[David Kastrup]

Jean Abou Samra  writes:

> Le 30/12/2022 à 16:59, Karlin High a écrit :
>> What are the chances of having a conversion tool that produces
>> LilyPond markup commands for the vectors in formats such as EPS?
>> Perhaps as an eps2ly or something. 
>
>
> Zero. PostScript is a full-fledged programming language. See the
> size of GhostScript, an implementation of PostScript; we're not
> going to create our own.

So?  ps2pdf exists.  It's not like it isn't already part of our toolkit,
so I don't see the problem with using it for creating a vector format.
pdftocairo can create an SVG from that, for example.

-- 
David Kastrup

Re: Missing items to make Cairo ready

[Lukas-Fabian Moser]

Am 30.12.22 um 17:34 schrieb Jean Abou Samra:

Le 30/12/2022 à 17:20, Lukas-Fabian Moser a écrit :
I agree with Jonas that it would be a considerable loss not to be 
able to import vector graphics. I don't concur with Jean's argument 
that one can replace them by using markup commands: One obvious use 
case for proper vector import would be including a publisher's logo, 
for instance.
A logo will occupy a small fraction of the page. Do you really need to 
have it in such high resolution that the file size becomes a problem?


In practice, probably. (Although there are also things like page-sized 
watermarks.)


But: One of the reasons to use LilyPond is the high-quality PDF output. 
I'm certain there are many users (myself included) who would regret 
having a raster graphic file lurking in their wonderful scalable PDF 
scores on principle.


That being said: My uneducated guess would be that with Cairo etc., 
it might be more natural/easy to import vector graphics in SVG format 
than in EPS?
Yes, I think. Basically, we could support embedding PDFs via Poppler, 
or SVGs via librsvg.
It would be nice to have both, but I really think at least one of them 
should work before the current backend (with its eps support) is abandoned.


Lukas

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 30/12/2022 à 17:20, Lukas-Fabian Moser a écrit :
I agree with Jonas that it would be a considerable loss not to be able 
to import vector graphics. I don't concur with Jean's argument that 
one can replace them by using markup commands: One obvious use case 
for proper vector import would be including a publisher's logo, for 
instance.



A logo will occupy a small fraction of the page. Do you really need to 
have it in such high resolution that the file size becomes a problem?



That being said: My uneducated guess would be that with Cairo etc., it 
might be more natural/easy to import vector graphics in SVG format 
than in EPS?


Yes, I think. Basically, we could support embedding PDFs via Poppler, or 
SVGs via librsvg.




OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Karlin High]

On Fri, Dec 30, 2022 at 10:20 AM Lukas-Fabian Moser  wrote:
> That being said: My uneducated guess would be that with Cairo etc., it
> might be more natural/easy to import vector graphics in SVG format than
> in EPS?

And apparently tools such as Inkscape offer EPS to SVG conversion.

<https://inkscape.org/forums/questions/using-inkscape-to-convert-from-eps-to-svg-same-result-as-adobe-illustrator/>

Although that reference mentions disagreements among different tools
with color space matching and such.
-- 
Karlin High
Missouri, USA

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 30/12/2022 à 16:59, Karlin High a écrit :

What are the chances of having a conversion tool that produces
LilyPond markup commands for the vectors in formats such as EPS?
Perhaps as an eps2ly or something. 



Zero. PostScript is a full-fledged programming language. See the
size of GhostScript, an implementation of PostScript; we're not
going to create our own.



OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Lukas-Fabian Moser]

Ok, in that case we can properly deprecate this once we have an
alternative (see above). I still think losing the ability to include
(external) vector graphics is a loss, though.

What are the chances of having a conversion tool that produces
LilyPond markup commands for the vectors in formats such as EPS?

Perhaps as an eps2ly or something.


I agree with Jonas that it would be a considerable loss not to be able 
to import vector graphics. I don't concur with Jean's argument that one 
can replace them by using markup commands: One obvious use case for 
proper vector import would be including a publisher's logo, for instance.


That being said: My uneducated guess would be that with Cairo etc., it 
might be more natural/easy to import vector graphics in SVG format than 
in EPS?


Lukas

Re: Missing items to make Cairo ready

[Karlin High]

On Fri, Dec 30, 2022 at 6:55 AM Jonas Hahnfeld via Discussions on
LilyPond development  wrote:
>
> On Fri, 2022-12-30 at 13:08 +0100, Jean Abou Samra wrote:
> > As I said, I don't see obvious use cases for \epsfile where you
> > really need vector graphics. LilyPond provides markup commands
> > for that.
>
> Ok, in that case we can properly deprecate this once we have an
> alternative (see above). I still think losing the ability to include
> (external) vector graphics is a loss, though.

What are the chances of having a conversion tool that produces
LilyPond markup commands for the vectors in formats such as EPS?

Perhaps as an eps2ly or something.
-- 
Karlin High
Missouri, USA

Re: Missing items to make Cairo ready

[Jonas Hahnfeld via Discussions on LilyPond development]

On Fri, 2022-12-30 at 13:08 +0100, Jean Abou Samra wrote:
> > I am not at all convinced by these workarounds just to get Cairo a
> > bit earlier and avoid walking the proper road to the clean solution
> > (ie properly deprecating what we don't want to or cannot support).
> 
> \postscript can be deprecated on other grounds, but we cannot
> really deprecate \epsfile without an offering an alternative,
> which means figuring out how to do PNGs via the default PS
> backend and GS.

Yes, I agree.

> I'm not really eager to do that, TBH.
> 
> > I think this is a bad limitation, and adding more dependencies into
> > the mix doesn't make it more appealing.
> 
> As I said, I don't see obvious use cases for \epsfile where you
> really need vector graphics. LilyPond provides markup commands
> for that.

Ok, in that case we can properly deprecate this once we have an
alternative (see above). I still think losing the ability to include
(external) vector graphics is a loss, though.

> > With Cairo, I am not aware of benefits for our users, it is more
> > motivated by internal maintenance considerations - please correct
> > me if I'm wrong.
> 
> - SVG rendering is *way* faster.
> 
> - It also embeds fonts, so the generated SVG files are device-
> independent.

Ok, this is SVG which we can probably agree that the backend is in a
bad state. At some point, there was a discussion on making Cairo the
default for producing SVGs but IIRC it faded out with the lack of
support for output-attributes, ie
https://gitlab.com/lilypond/lilypond/-/issues/6503

Once that is solved, only enabling Cairo by default for SVGs may be a
separate discussion that is worth having.

> - PDF rendering is a little faster.
> 
> - It does not use a subprocess, which makes it easier to sandbox it
>    (cf. the problem with LilyPond on GitHub actions that was
> discussed some time ago on this list).
>    (OK, this is defeated by the "workaround" as you noticed.)

Both actually, right?

> - It makes it easier for developers to add certain features
>    that would be harder to support in the current backends,
>    like PNG, which benefit users in the end.

"Developer happiness" is hard to sell to users unless it results in
immediate benefits; I don't think it's a convincing argument at the
moment.


signature.asc
Description: This is a digitally signed message part

Re: Missing items to make Cairo ready

[Jean Abou Samra]

I am not at all convinced by these workarounds just to get Cairo a bit
earlier and avoid walking the proper road to the clean solution (ie
properly deprecating what we don't want to or cannot support).



\postscript can be deprecated on other grounds, but we cannot
really deprecate \epsfile without an offering an alternative,
which means figuring out how to do PNGs via the default PS
backend and GS. I'm not really eager to do that, TBH.



I think this is a bad limitation, and adding more dependencies into the
mix doesn't make it more appealing.




As I said, I don't see obvious use cases for \epsfile where you
really need vector graphics. LilyPond provides markup commands
for that.




With Cairo, I am not aware of benefits for our users, it is more
motivated by internal maintenance considerations - please correct me if
I'm wrong.



- SVG rendering is *way* faster.

- It also embeds fonts, so the generated SVG files are device-independent.

- PDF rendering is a little faster.

- It does not use a subprocess, which makes it easier to sandbox it
  (cf. the problem with LilyPond on GitHub actions that was discussed
  some time ago on this list).
  (OK, this is defeated by the "workaround" as you noticed.)

- It makes it easier for developers to add certain features
  that would be harder to support in the current backends,
  like PNG, which benefit users in the end.




OpenPGP_signature
Description: OpenPGP digital signature

Re: Missing items to make Cairo ready

[Jonas Hahnfeld via Discussions on LilyPond development]

On Thu, 2022-12-29 at 22:55 +0100, Jean Abou Samra wrote:
> Le 29/12/2022 à 12:19, Jonas Hahnfeld a écrit :
> > (FWIW I don't think it is a good idea to add features that will only
> > work with Cairo. That sounds like a recipe for confusion to me, but I
> > haven't yet had the time to properly look into the merge request...)
> 
> 
> I don't really see what the problem is.
> 
> If you try to use a PNG image in the default PS backend, you
> get a user-friendly warning explaining that you need to run with
> -dbackend=cairo (OK, makes me think I should add this to the default
> SVG backend as well).
> 
> It's not much different from \epsfile being supported in the
> default PS backend but not in the default SVG backend.

It is: \epsfile works with the default PS backend producing PDFs and
PNGs aka what probably 90% of the users care about; the new \image with
PNGs *only* works with the non-default Cairo backend. That is a huge
difference for me.


> PNG embedding should be feasible for the default PS and SVG
> backends too, but I'm not willing to spend time on it.
> 
> 
> > > So, to support \epsfile
> > > and \postscript, we can convert and embed as PNG.
> > > Of course, this means we cannot drop Ghostscript,
> > > but we're not going to do it anytime soon anyway.
> > PNG is a pixel-format, so depending on what users embed images for,
> > this may result in very inferior quality or huge files (if you render
> > at insane dimensions).
> 
> 
> Well, if GS is still around, there is the other route too: create
> PS output via Cairo, and convert to PDF with GS like we're already
> doing. That way, you can have vector graphics into PDF from EPS.

Would we want to go the Cairo -> GS route by default? Then we lose one
big advantage that we wanted to gain from Cairo, keeping everything in-
process. If we don't, we have different output paths depending on
whether the user has \epsfile in the input file, which sounds like a
nightmare to support.

I am not at all convinced by these workarounds just to get Cairo a bit
earlier and avoid walking the proper road to the clean solution (ie
properly deprecating what we don't want to or cannot support).


> That said, for this to be a problem, you need to have an EPS file
> that actually contains vector graphics. My impression is that in
> most cases, users pass files that contain embedded raster graphics,
> probably converted from PNG, JPEG or TIFF in the first place.
> 
> > Which raises another question: How do you embed
> > vector graphics into Cairo?
> 
> Cairo itself doesn't understand PS or PDF (it can just
> embed the EPS content into EPS/PS without actually parsing
> it). For that, I think you would have to use Poppler,
> which has a Cairo backend.

I think this is a bad limitation, and adding more dependencies into the
mix doesn't make it more appealing.


> > I don't agree here. It is hard to sell any migration without some level
> > of parity. We had this with Python 3 and Guile 2, and I think it is
> > even more important for the output backend.
> 
> Sorry, I don't understand what you mean by this. Can
> you elaborate?

The two big transitions that I have been part of tried to cause as
little problems to users as possible: For Python 3, it was hopefully
transparent. All I am aware of were problems found in some support
scripts long after the transition, and those were easily fixed.

For Guile 2.2, all user scores should continue to work and performance
should be comparable. It is true that Scheme library developers had to
migrate code, but in exchange they get proper Unicode support. Overall
the transition finally brought 64-bit binaries on macOS and Windows,
which is significant incentive for users to upgrade.

With Cairo, I am not aware of benefits for our users, it is more
motivated by internal maintenance considerations - please correct me if
I'm wrong. In order to "sell" this to users, we must at the very least
not make their experience worse in any regard. Otherwise I fear that
part of the community will stick with an older version because it is
superior in their eyes. IMHO we must avoid this situation as hard as we
can. (From my point of view we had a similar issue, at least until very
recently, with 2.18.2 simply because it was around for "too long".)

Jonas


signature.asc
Description: This is a digitally signed message part

Re: Missing items to make Cairo ready

[Jean Abou Samra]

Le 29/12/2022 à 12:19, Jonas Hahnfeld a écrit :

(FWIW I don't think it is a good idea to add features that will only
work with Cairo. That sounds like a recipe for confusion to me, but I
haven't yet had the time to properly look into the merge request...)



I don't really see what the problem is.

If you try to use a PNG image in the default PS backend, you
get a user-friendly warning explaining that you need to run with
-dbackend=cairo (OK, makes me think I should add this to the default
SVG backend as well).

It's not much different from \epsfile being supported in the
default PS backend but not in the default SVG backend.

PNG embedding should be feasible for the default PS and SVG
backends too, but I'm not willing to spend time on it.



So, to support \epsfile
and \postscript, we can convert and embed as PNG.
Of course, this means we cannot drop Ghostscript,
but we're not going to do it anytime soon anyway.

PNG is a pixel-format, so depending on what users embed images for,
this may result in very inferior quality or huge files (if you render
at insane dimensions).



Well, if GS is still around, there is the other route too: create
PS output via Cairo, and convert to PDF with GS like we're already
doing. That way, you can have vector graphics into PDF from EPS.

That said, for this to be a problem, you need to have an EPS file
that actually contains vector graphics. My impression is that in
most cases, users pass files that contain embedded raster graphics,
probably converted from PNG, JPEG or TIFF in the first place.




Which raises another question: How do you embed
vector graphics into Cairo?



Cairo itself doesn't understand PS or PDF (it can just
embed the EPS content into EPS/PS without actually parsing
it). For that, I think you would have to use Poppler,
which has a Cairo backend.




I don't agree here. It is hard to sell any migration without some level
of parity. We had this with Python 3 and Guile 2, and I think it is
even more important for the output backend.



Sorry, I don't understand what you mean by this. Can
you elaborate?



OpenPGP_signature
Description: OpenPGP digital signature