Re: Output PDF to stdout
Hello Vlad,
You will not avoid intermediate files in any case, as Lilypond with gs output
will generate ps code, which is then turned into a pdf. If you want to pipe
PDFs for convenience you might create a wrapper say plilypond in the search
path doing something like this:
#!/bin/bash
t=$(mktemp) 1>&2
lilypond -o "$t" ${@} 1>&2
cat "$t".pdf
rm "$t".pdf 1>&2
Cheers,
Valentin
Am Donnerstag, 4. Jänner 2024, 23:34:28 CET schrieb Volodymyr Prokopyuk:
> Hi,
>
> I know that lilypond can receive a source.ly file from the stdin by
> using lilypond
> -. Is it possible for lilypond to output PDF to the stdout?
>
> My motivation behind using lilypond in a pipeline is to speed up PDF
> generation by avoiding storing intermediary files on disk. The pipeline I'd
> like to implement is
> cat source.ly | lintLy | lilypond - | optimizePDF > score.pdf
> Currently lilypond engraves PDF on disk, so a PDF optimizer has to read,
> optimize, and substitute PDF on disk.
>
> Thank you,
> Vlad
signature.asc
Description: This is a digitally signed message part.
Re: Output PDF to stdout
On 13 Jan 2024, at 02:05, Curt McDowell wrote:Have you actually tried this? LilyPond appends ".pdf" to the output filename (and ".midi"). If you try to make the fifo ending in ".pdf", you find lilypond removes the fifo before writing a new file. For the same reasons, the output file cannot be /dev/stdout. Maybe you could write something exciting using FUSE or Docker.On 1/7/2024 5:41 AM, Raphael Mankin wrote:On 05/01/2024 03:46, David Wright wrote:Alternatively, use a fifo, e.g. fifo=/var/tmp/ly-pdf$$rm -rf $fifomknod $fifo pcat source.ly | lintLy | lilypond -o $fifo &optimizePDF < $fifo > score.pdfwaitrm -rf $fifoA trick I learned when doing Oracle backups. It also would not write to stdout.As an offtopic sidenote: even if it worked/works, according to the man-pages of mknod mkfifo should be preferred over mknod to create the FIFO."POSIX.1-2001 says: "The only portable use of mknod() is to create a FIFO-special file. If mode is not S_IFIFO or dev is not 0, the behavior of mknod() is unspecified." However, nowadays one should never use mknod() for this purpose; one should use mkfifo(3), a function especially defined for this purpose."
Re: Output PDF to stdout
Have you actually tried this? LilyPond appends ".pdf" to the output filename (and ".midi"). If you try to make the fifo ending in ".pdf", you find lilypond removes the fifo before writing a new file. For the same reasons, the output file cannot be /dev/stdout. Maybe you could write something exciting using FUSE or Docker. On 1/7/2024 5:41 AM, Raphael Mankin wrote: On 05/01/2024 03:46, David Wright wrote: Alternatively, use a fifo, e.g. fifo=/var/tmp/ly-pdf$$ rm -rf $fifo mknod $fifo p cat source.ly | lintLy | lilypond -o $fifo & optimizePDF < $fifo > score.pdf wait rm -rf $fifo A trick I learned when doing Oracle backups. It also would not write to stdout.
Re: Output PDF to stdout
On 05/01/2024 03:46, David Wright wrote:
On Thu 04 Jan 2024 at 23:34:28 (+0100), Volodymyr Prokopyuk wrote:
I know that lilypond can receive a source.ly file from the stdin by
using lilypond
-. Is it possible for lilypond to output PDF to the stdout?
My motivation behind using lilypond in a pipeline is to speed up PDF
generation by avoiding storing intermediary files on disk. The pipeline I'd
like to implement is
cat source.ly | lintLy | lilypond - | optimizePDF > score.pdf
Currently lilypond engraves PDF on disk, so a PDF optimizer has to read,
optimize, and substitute PDF on disk.
You could create a RAM disk to hold your intermediate files,
something along the lines of:
Ramdir="/dev/shm/${FUNCNAME[0]}$(printf '%(%s)T' -1)"
$ mkdir -p "$Ramdir"
The first line (which I run in a bash function, for a quite different
purpose) creates a pseudounique but recognizable name, and the second
line creates the directory.
Cheers,
David.
Alternatively, use a fifo, e.g.
fifo=/var/tmp/ly-pdf$$
rm -rf $fifo
mknod $fifo p
cat source.ly | lintLy | lilypond -o $fifo &
optimizePDF < $fifo > score.pdf
wait
rm -rf $fifo
A trick I learned when doing Oracle backups. It also would not write to
stdout.
--
Political correctness: a kind of McCarthyite movement in reverse which,
in the name of tolerance proscribes all reference to gender, ethnicity,
color of skin, sexual preference, social provenance and even age. It has
no leaders, as far as I am aware, only terrified disciples. - John le Carre
Re: Output PDF to stdout
Thank you very much for your thoughts!
I expected that the PDF output to stdout would be difficult to implement
due to multiple Lilypond backends, and, probably, is not a real bottleneck
in PDF generation. It is nice to know that intermediary files are already
handled by a FS cache, and can be externally optimized by using a RAM disk.
Thank you,
Vlad
On Fri, Jan 5, 2024 at 4:46 AM David Wright
wrote:
> On Thu 04 Jan 2024 at 23:34:28 (+0100), Volodymyr Prokopyuk wrote:
> >
> > I know that lilypond can receive a source.ly file from the stdin by
> > using lilypond
> > -. Is it possible for lilypond to output PDF to the stdout?
> >
> > My motivation behind using lilypond in a pipeline is to speed up PDF
> > generation by avoiding storing intermediary files on disk. The pipeline
> I'd
> > like to implement is
> > cat source.ly | lintLy | lilypond - | optimizePDF > score.pdf
> > Currently lilypond engraves PDF on disk, so a PDF optimizer has to read,
> > optimize, and substitute PDF on disk.
>
> You could create a RAM disk to hold your intermediate files,
> something along the lines of:
>
>Ramdir="/dev/shm/${FUNCNAME[0]}$(printf '%(%s)T' -1)"
> $ mkdir -p "$Ramdir"
>
> The first line (which I run in a bash function, for a quite different
> purpose) creates a pseudounique but recognizable name, and the second
> line creates the directory.
>
> Cheers,
> David.
>
Re: Output PDF to stdout
On Thu 04 Jan 2024 at 23:34:28 (+0100), Volodymyr Prokopyuk wrote:
>
> I know that lilypond can receive a source.ly file from the stdin by
> using lilypond
> -. Is it possible for lilypond to output PDF to the stdout?
>
> My motivation behind using lilypond in a pipeline is to speed up PDF
> generation by avoiding storing intermediary files on disk. The pipeline I'd
> like to implement is
> cat source.ly | lintLy | lilypond - | optimizePDF > score.pdf
> Currently lilypond engraves PDF on disk, so a PDF optimizer has to read,
> optimize, and substitute PDF on disk.
You could create a RAM disk to hold your intermediate files,
something along the lines of:
Ramdir="/dev/shm/${FUNCNAME[0]}$(printf '%(%s)T' -1)"
$ mkdir -p "$Ramdir"
The first line (which I run in a bash function, for a quite different
purpose) creates a pseudounique but recognizable name, and the second
line creates the directory.
Cheers,
David.
Re: Output PDF to stdout
On Fri, 5 Jan 2024, Dr. Arne Babenhauserheide wrote: > > intermediate "files" will be written to and read from the buffer cache at > > RAM speed and only later go to the disk in the background. > > That depends on the file system and its synchronization model. I once > sped up a script to control a cluster by more than factor 10 by avoiding > intermediate writes to the shared network filesystem of the cluster > nodes. That made the difference between 15 minutes and 30 seconds > runtime. I'm sure you're already aware of the important differences between that and the situation described in this thread. -- Matthew Skala msk...@ansuz.sooke.bc.ca People before tribes. https://ansuz.sooke.bc.ca/
Re: Output PDF to stdout
msk...@ansuz.sooke.bc.ca writes: > On Thu, 4 Jan 2024, Volodymyr Prokopyuk wrote: > >> My motivation behind using lilypond in a pipeline is to speed up PDF >> generation by avoiding storing intermediary files on disk. The pipeline I'd > > Is that issue real? In Linux and most other operating systems, > intermediate "files" will be written to and read from the buffer cache at > RAM speed and only later go to the disk in the background. That depends on the file system and its synchronization model. I once sped up a script to control a cluster by more than factor 10 by avoiding intermediate writes to the shared network filesystem of the cluster nodes. That made the difference between 15 minutes and 30 seconds runtime. Best wishes, Arne -- Unpolitisch sein heißt politisch sein, ohne es zu merken. draketo.de signature.asc Description: PGP signature
Re: Output PDF to stdout
This doesn't exist, sorry. It wouldn't be *that* straightforward to implement, because there are multiple code paths for output (GhostScript via API, GhostScript via subprocess, SVG backend and Cairo). I'm not even sure that direct PDF output in GhostScript's PS → PDF conversion is not going through an intermediate file under the hood. Either way, I really doubt this would make a significant speed difference. signature.asc Description: This is a digitally signed message part
Re: Output PDF to stdout
On Thu, 4 Jan 2024, Volodymyr Prokopyuk wrote: > My motivation behind using lilypond in a pipeline is to speed up PDF > generation by avoiding storing intermediary files on disk. The pipeline I'd Is that issue real? In Linux and most other operating systems, intermediate "files" will be written to and read from the buffer cache at RAM speed and only later go to the disk in the background. The subsequent process that uses the file won't be waiting for the disk I/O to complete. Even if a pipe might theoretically be even faster than the buffer cache, the intermediate unoptimized PDF is unlikely to be the bottleneck step in generation of an optimized PDF. -- Matthew Skala msk...@ansuz.sooke.bc.ca People before tribes. https://ansuz.sooke.bc.ca/
Output PDF to stdout
Hi, I know that lilypond can receive a source.ly file from the stdin by using lilypond -. Is it possible for lilypond to output PDF to the stdout? My motivation behind using lilypond in a pipeline is to speed up PDF generation by avoiding storing intermediary files on disk. The pipeline I'd like to implement is cat source.ly | lintLy | lilypond - | optimizePDF > score.pdf Currently lilypond engraves PDF on disk, so a PDF optimizer has to read, optimize, and substitute PDF on disk. Thank you, Vlad
