Re: [flac-dev] libFLAC optimizations request

[Gabriel Corneanu]

Hi,

On Sun, Feb 4, 2018 at 11:50 PM, Brian Willoughby
 wrote:
> I wasn’t suggesting that you run metaflac, but that you examine its source to 
> see how it creates new FLAC files without the Vorbis comment. As far as I 
> know, metaflac uses the standard libFLAC and creates files without the Vorbis 
> overhead.

Without reading the sources, I simply assume that metaflac is parsing
the metadata blocks and filtering as required. Of course it is
possible with libFLAC, but the overhead is huge (parse stream, filter,
save back, etc) compared to simply not generating it.

> My quick review of the source seemed to indicate that calling 
> FLAC__metadata_object_new(FLAC__METADATA_TYPE_VORBIS_COMMENT) will create the 
> comment, but I assumed that you could pass in NULL. I mistakenly thought that 
> was FLAC__stream_encoder_new().

The comment is auto-generated in "init_stream_internal_", called from
all _init functions. I am NOT explicitly creating any metadata object.

> There is a comment saying that the Ogg FLAC mapping requires a 
> VORBIS_COMMENT, but that doesn’t necessarily mean that standard FLAC also 
> requires the Vorbis comment.

That's what I thought, maybe if the target is NOT Ogg we could by
default not generate the comment; the user can still explicitly
generate it.

> Looking around metaflac sources, I see 
> FLAC__metadata_iterator_delete_block(iterator, options->use_padding), which 
> might be the code responsible for removing the Vorbis comment.
>
> In other words, perhaps there is no existing way to avoid having the encoder 
> create the Vorbis comment metadata, but there might be a way to delete that 
> particular metadata before the file is saved. A closer look at how metaflac 
> creates a file without Vorbis comments should help.
>
> Brian
>

I will create a PR with my suggestions.

Gabriel
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] libFLAC optimizations request

[Brian Willoughby]

I wasn’t suggesting that you run metaflac, but that you examine its source to 
see how it creates new FLAC files without the Vorbis comment. As far as I know, 
metaflac uses the standard libFLAC and creates files without the Vorbis 
overhead.

My quick review of the source seemed to indicate that calling 
FLAC__metadata_object_new(FLAC__METADATA_TYPE_VORBIS_COMMENT) will create the 
comment, but I assumed that you could pass in NULL. I mistakenly thought that 
was FLAC__stream_encoder_new().

There is a comment saying that the Ogg FLAC mapping requires a VORBIS_COMMENT, 
but that doesn’t necessarily mean that standard FLAC also requires the Vorbis 
comment.

Looking around metaflac sources, I see 
FLAC__metadata_iterator_delete_block(iterator, options->use_padding), which 
might be the code responsible for removing the Vorbis comment.

In other words, perhaps there is no existing way to avoid having the encoder 
create the Vorbis comment metadata, but there might be a way to delete that 
particular metadata before the file is saved. A closer look at how metaflac 
creates a file without Vorbis comments should help.

Brian


On Feb 4, 2018, at 6:23 AM, Gabriel Corneanu  wrote:
> The problem is really as I wrote:
> 1. Metaflac is no option for me, I use libFLAC.dll
> 2. There is no way (at least how I read the code) to avoid saving
> comment with libFLAC; I would appreciate an extra option to avoid it,
> which can default to old behavior if compatibility is important.
> 3. I have a high speed application, where re-initializing an encoder
> is really significant. On corner cases it causes an 25% overhead! Of
> course I don't expect it to be that significant in normal cases.
> 
> Thanks for all replies, but I don't have the code at home.
> I will create a patch with my changes for review.
> 
> Regards,
> Gabriel
> 
> On Sun, Feb 4, 2018 at 9:57 AM, Brian Willoughby  
> wrote:
>> Correction, the flac command-line does create a 40-byte Vorbis comment by 
>> default. I just never noticed it before. I’ve been using —no-padding all 
>> these years for minimal file sizes without realizing that I could save 
>> another 40 bytes.
>> 
>> Anyway, since metaflac can remove the Vorbis comment using the standard 
>> library, then you should be able to create a solution without modifying 
>> libFLAC.
>> 
>> On Feb 4, 2018, at 12:43 AM, Brian Willoughby  
>> wrote:
>>> Gabriel,
>>> 
>>> metadata_has_vorbis_comment is a FLAC__bool which defaults to false. There 
>>> should be no reason to modify stream_encoder.c, but just modify the caller.
>>> 
>>> The following command:
>>> 
>>> metaflac —remove —block-type=VORBIS_COMMENT —don’t-use-padding
>>> 
>>> … will remove Vorbis comments from existing files, so is must be legal 
>>> without modifying the library. metaflac can clearly create a new FLAC file 
>>> without the Vorbis comment.
>>> 
>>> I use the flac command-line, and I never get Vorbis comments in the files 
>>> that I create. Perhaps you are using another tool which assumes Vorbis 
>>> comments are needed.
>>> 
>>> 
>>> The FLAC algorithm is not dependent upon sample rate. AIFF has an 80-bit 
>>> floating point type for sample rate, so it should be able to handle 40 MHz. 
>>> I assume that any AIFF can be converted to FLAC losslessly, but I have not 
>>> tested whether certain sample rates are rejected. FLAC itself only supports 
>>> sample rates up to 655,350 Hz, so you may have a problem there unless you 
>>> “lie” about the sample rate when creating the file. Maybe you could just 
>>> establish a private convention to divide the sample rate by 100 to make 
>>> yours fit. 40 MHz would map to 400 kHz, 10 MHz would map to 100 kHz, and 5 
>>> MHz would map to 50 kHz.
>>> 
>>> 
>>> You’re probably asking for trouble if you try to reuse an encoder. It seems 
>>> like there would always be some risk that details from the previous file 
>>> would bleed through into the next. Have you benchmarked allocation and 
>>> initialization? Is it really that slow? In order to reuse an encoder, 
>>> you’ll need to overwrite all state variables, and I don’t see how that 
>>> could be much faster than simply allocating them anew. Perhaps you could 
>>> allocate groups of encoders at once, if that would speed the process.
>>> 
>>> 
>>> On Feb 1, 2018, at 4:29 AM, Gabriel Corneanu  
>>> wrote:
 Hello all
 
 I am using libFLAC in a corner application, compressing a lot of small 
 signals.
 First is a general question: in our application we have signals in range 
 5-10 MHz, potentially 40MHz! Is there any potential problem with that?? 
 The mac sample rate is limited in flac, but it doesn't really seem to be a 
 problem.
 The output is stored as blob in a sqlite database, it never needs to be a 
 valid audio file outside our application.
 In my tests, the signals are compressed very well, much 

Re: [flac-dev] libFLAC optimizations request

[Gabriel Corneanu]

The problem is really as I wrote:
1. Metaflac is no option for me, I use libFLAC.dll
2. There is no way (at least how I read the code) to avoid saving
comment with libFLAC; I would appreciate an extra option to avoid it,
which can default to old behavior if compatibility is important.
3. I have a high speed application, where re-initializing an encoder
is really significant. On corner cases it causes an 25% overhead! Of
course I don't expect it to be that significant in normal cases.

Thanks for all replies, but I don't have the code at home.
I will create a patch with my changes for review.

Regards,
Gabriel

On Sun, Feb 4, 2018 at 9:57 AM, Brian Willoughby
 wrote:
> Correction, the flac command-line does create a 40-byte Vorbis comment by 
> default. I just never noticed it before. I’ve been using —no-padding all 
> these years for minimal file sizes without realizing that I could save 
> another 40 bytes.
>
> Anyway, since metaflac can remove the Vorbis comment using the standard 
> library, then you should be able to create a solution without modifying 
> libFLAC.
>
> Brian
>
>
> On Feb 4, 2018, at 12:43 AM, Brian Willoughby  wrote:
>> Gabriel,
>>
>> metadata_has_vorbis_comment is a FLAC__bool which defaults to false. There 
>> should be no reason to modify stream_encoder.c, but just modify the caller.
>>
>> The following command:
>>
>> metaflac —remove —block-type=VORBIS_COMMENT —don’t-use-padding
>>
>> … will remove Vorbis comments from existing files, so is must be legal 
>> without modifying the library. metaflac can clearly create a new FLAC file 
>> without the Vorbis comment.
>>
>> I use the flac command-line, and I never get Vorbis comments in the files 
>> that I create. Perhaps you are using another tool which assumes Vorbis 
>> comments are needed.
>>
>>
>> The FLAC algorithm is not dependent upon sample rate. AIFF has an 80-bit 
>> floating point type for sample rate, so it should be able to handle 40 MHz. 
>> I assume that any AIFF can be converted to FLAC losslessly, but I have not 
>> tested whether certain sample rates are rejected. FLAC itself only supports 
>> sample rates up to 655,350 Hz, so you may have a problem there unless you 
>> “lie” about the sample rate when creating the file. Maybe you could just 
>> establish a private convention to divide the sample rate by 100 to make 
>> yours fit. 40 MHz would map to 400 kHz, 10 MHz would map to 100 kHz, and 5 
>> MHz would map to 50 kHz.
>>
>>
>> You’re probably asking for trouble if you try to reuse an encoder. It seems 
>> like there would always be some risk that details from the previous file 
>> would bleed through into the next. Have you benchmarked allocation and 
>> initialization? Is it really that slow? In order to reuse an encoder, you’ll 
>> need to overwrite all state variables, and I don’t see how that could be 
>> much faster than simply allocating them anew. Perhaps you could allocate 
>> groups of encoders at once, if that would speed the process.
>>
>>
>> On Feb 1, 2018, at 4:29 AM, Gabriel Corneanu  
>> wrote:
>>> Hello all
>>>
>>> I am using libFLAC in a corner application, compressing a lot of small 
>>> signals.
>>> First is a general question: in our application we have signals in range 
>>> 5-10 MHz, potentially 40MHz! Is there any potential problem with that?? The 
>>> mac sample rate is limited in flac, but it doesn't really seem to be a 
>>> problem.
>>> The output is stored as blob in a sqlite database, it never needs to be a 
>>> valid audio file outside our application.
>>> In my tests, the signals are compressed very well, much better than general 
>>> compression libraries like zlib, zstd, etc.
>>>
>>> Now other small issues; I also made some tickets about them, but I thought 
>>> asking here might be better.
>>>
>>> 1. I would like to avoid saving vorbis comment, by default ~40 bytes. Right 
>>> now the only option is to modify stream_encoder.c, see 
>>> "metadata_has_vorbis_comment".
>>>
>>> 2. Speed is very important, therefore I would like to reuse an encoder 
>>> without re-initializing everything.
>>> Ideally I would like 2 (exported) functions: "flush" and "restart".
>>> "Flush" is self-explanatory, should properly end the encoding. I could 
>>> split myself "flush" from "finish", it looks relatively simple.
>>> "Restart" should keep all current settings, generate a new stream header 
>>> and clear everything for encoding a new signal.
>>> It' clear that current settings, re-creating windows, cpu-dependent 
>>> functions, etc could be kept around.
>>> I was not quickly able to extract all the necessary initialization from 
>>> "init_stream_internal_" into a new "FLAC__stream_encoder_restart" function.
>>>
>>> Regards,
>>> Gabriel Corneanu
>
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] libFLAC optimizations request

[Brian Willoughby]

Correction, the flac command-line does create a 40-byte Vorbis comment by 
default. I just never noticed it before. I’ve been using —no-padding all these 
years for minimal file sizes without realizing that I could save another 40 
bytes.

Anyway, since metaflac can remove the Vorbis comment using the standard 
library, then you should be able to create a solution without modifying libFLAC.

Brian


On Feb 4, 2018, at 12:43 AM, Brian Willoughby  wrote:
> Gabriel,
> 
> metadata_has_vorbis_comment is a FLAC__bool which defaults to false. There 
> should be no reason to modify stream_encoder.c, but just modify the caller.
> 
> The following command:
> 
> metaflac —remove —block-type=VORBIS_COMMENT —don’t-use-padding
> 
> … will remove Vorbis comments from existing files, so is must be legal 
> without modifying the library. metaflac can clearly create a new FLAC file 
> without the Vorbis comment.
> 
> I use the flac command-line, and I never get Vorbis comments in the files 
> that I create. Perhaps you are using another tool which assumes Vorbis 
> comments are needed.
> 
> 
> The FLAC algorithm is not dependent upon sample rate. AIFF has an 80-bit 
> floating point type for sample rate, so it should be able to handle 40 MHz. I 
> assume that any AIFF can be converted to FLAC losslessly, but I have not 
> tested whether certain sample rates are rejected. FLAC itself only supports 
> sample rates up to 655,350 Hz, so you may have a problem there unless you 
> “lie” about the sample rate when creating the file. Maybe you could just 
> establish a private convention to divide the sample rate by 100 to make yours 
> fit. 40 MHz would map to 400 kHz, 10 MHz would map to 100 kHz, and 5 MHz 
> would map to 50 kHz.
> 
> 
> You’re probably asking for trouble if you try to reuse an encoder. It seems 
> like there would always be some risk that details from the previous file 
> would bleed through into the next. Have you benchmarked allocation and 
> initialization? Is it really that slow? In order to reuse an encoder, you’ll 
> need to overwrite all state variables, and I don’t see how that could be much 
> faster than simply allocating them anew. Perhaps you could allocate groups of 
> encoders at once, if that would speed the process.
> 
> 
> On Feb 1, 2018, at 4:29 AM, Gabriel Corneanu  
> wrote:
>> Hello all
>> 
>> I am using libFLAC in a corner application, compressing a lot of small 
>> signals.
>> First is a general question: in our application we have signals in range 
>> 5-10 MHz, potentially 40MHz! Is there any potential problem with that?? The 
>> mac sample rate is limited in flac, but it doesn't really seem to be a 
>> problem.
>> The output is stored as blob in a sqlite database, it never needs to be a 
>> valid audio file outside our application.
>> In my tests, the signals are compressed very well, much better than general 
>> compression libraries like zlib, zstd, etc.
>> 
>> Now other small issues; I also made some tickets about them, but I thought 
>> asking here might be better.
>> 
>> 1. I would like to avoid saving vorbis comment, by default ~40 bytes. Right 
>> now the only option is to modify stream_encoder.c, see 
>> "metadata_has_vorbis_comment".
>> 
>> 2. Speed is very important, therefore I would like to reuse an encoder 
>> without re-initializing everything.
>> Ideally I would like 2 (exported) functions: "flush" and "restart".
>> "Flush" is self-explanatory, should properly end the encoding. I could split 
>> myself "flush" from "finish", it looks relatively simple.
>> "Restart" should keep all current settings, generate a new stream header and 
>> clear everything for encoding a new signal.
>> It' clear that current settings, re-creating windows, cpu-dependent 
>> functions, etc could be kept around. 
>> I was not quickly able to extract all the necessary initialization from 
>> "init_stream_internal_" into a new "FLAC__stream_encoder_restart" function.
>> 
>> Regards,
>> Gabriel Corneanu

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] libFLAC optimizations request

[Brian Willoughby]

Gabriel,

metadata_has_vorbis_comment is a FLAC__bool which defaults to false. There 
should be no reason to modify stream_encoder.c, but just modify the caller.

The following command:

metaflac —remove —block-type=VORBIS_COMMENT —don’t-use-padding

… will remove Vorbis comments from existing files, so is must be legal without 
modifying the library. metaflac can clearly create a new FLAC file without the 
Vorbis comment.

I use the flac command-line, and I never get Vorbis comments in the files that 
I create. Perhaps you are using another tool which assumes Vorbis comments are 
needed.


The FLAC algorithm is not dependent upon sample rate. AIFF has an 80-bit 
floating point type for sample rate, so it should be able to handle 40 MHz. I 
assume that any AIFF can be converted to FLAC losslessly, but I have not tested 
whether certain sample rates are rejected. FLAC itself only supports sample 
rates up to 655,350 Hz, so you may have a problem there unless you “lie” about 
the sample rate when creating the file. Maybe you could just establish a 
private convention to divide the sample rate by 100 to make yours fit. 40 MHz 
would map to 400 kHz, 10 MHz would map to 100 kHz, and 5 MHz would map to 50 
kHz.


You’re probably asking for trouble if you try to reuse an encoder. It seems 
like there would always be some risk that details from the previous file would 
bleed through into the next. Have you benchmarked allocation and 
initialization? Is it really that slow? In order to reuse an encoder, you’ll 
need to overwrite all state variables, and I don’t see how that could be much 
faster than simply allocating them anew. Perhaps you could allocate groups of 
encoders at once, if that would speed the process.


On Feb 1, 2018, at 4:29 AM, Gabriel Corneanu  wrote:
> Hello all
> 
> I am using libFLAC in a corner application, compressing a lot of small 
> signals.
> First is a general question: in our application we have signals in range 5-10 
> MHz, potentially 40MHz! Is there any potential problem with that?? The mac 
> sample rate is limited in flac, but it doesn't really seem to be a problem.
> The output is stored as blob in a sqlite database, it never needs to be a 
> valid audio file outside our application.
> In my tests, the signals are compressed very well, much better than general 
> compression libraries like zlib, zstd, etc.
> 
> Now other small issues; I also made some tickets about them, but I thought 
> asking here might be better.
> 
> 1. I would like to avoid saving vorbis comment, by default ~40 bytes. Right 
> now the only option is to modify stream_encoder.c, see 
> "metadata_has_vorbis_comment".
> 
> 2. Speed is very important, therefore I would like to reuse an encoder 
> without re-initializing everything.
> Ideally I would like 2 (exported) functions: "flush" and "restart".
> "Flush" is self-explanatory, should properly end the encoding. I could split 
> myself "flush" from "finish", it looks relatively simple.
> "Restart" should keep all current settings, generate a new stream header and 
> clear everything for encoding a new signal.
> It' clear that current settings, re-creating windows, cpu-dependent 
> functions, etc could be kept around. 
> I was not quickly able to extract all the necessary initialization from 
> "init_stream_internal_" into a new "FLAC__stream_encoder_restart" function.
> 
> Regards,
> Gabriel Corneanu

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] libFLAC optimizations request

[Erik de Castro Lopo]

Gabriel Corneanu wrote:

> Is this interesting for the project?

I would theoretically accept a patch that allows direct access to the
functionality you seek, as long as it doesn't change the existing API.

> If yes, how should I share it? As a patch against current git?

Probably easiest as as PR against:

   https://github.com/xiph/flac/

> 
> Other small details:
> - I commented out by default the vorbis comment; if this is important
> to be kept, I would like an option (e.g. I could save ~100MB out of
> ~1.1GB compressed data).

I'm  pretty sure this default vorbis comment is part of the official
specfication of FLAC and hence should probably not even be made optional.
You can probably avoid it by using the library directly instead of using
the flac command line program.

Erik
-- 
--
Erik de Castro Lopo
http://www.mega-nerd.com/
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] libFLAC optimizations request

[Erik de Castro Lopo]

Gabriel Corneanu wrote:

> I am using libFLAC in a corner application, compressing *a lot* of small
> signals.

You are of course aware that this is *not* what FLAC was designed for,

> First is a general question: in our application we have signals in range
> 5-10 MHz, potentially 40MHz! Is there any potential problem with that??

No idea whatsoever. It really depends on the signal. FLAC is *designed*
and optimised for compressing audio signals, specifically music. For
cases where the FLAC compressed signal is does not actually compress
well (on example is white noise) the audio is included uncompressed.

> The
> mac sample rate is limited in flac, but it doesn't really seem to be a
> problem.

FLAC is the Free Lossless **AUDIO** codec.

> The output is stored as blob in a sqlite database, it *never *needs to be a
> valid audio file outside our application.
> In my tests, the signals are compressed very well, much better than general
> compression libraries like zlib, zstd, etc.

So regardless of the higher sample rate your signals seem to be enough
like audio for FLAC to compress them well.

> Now other small issues; I also made some tickets about them, but I thought
> asking here might be better.
> 
> 1. I would like to avoid saving vorbis comment, by default ~40 bytes. Right
> now the only option is to modify stream_encoder.c, see
> "metadata_has_vorbis_comment".

Are you using the FLAC library directly or the flac command line program?

If you are using the flac command line program, please feel free to hack
that to your hearts content (as long as you abide by the very libreral
license). 

> 2. Speed is very important, therefore I would like to reuse an encoder
> without re-initializing everything.
> Ideally I would like 2 (exported) functions: "flush" and "restart".
> "Flush" is self-explanatory, should properly end the encoding. I could
> split myself "flush" from "finish", it looks relatively simple.
> "Restart" should keep all current settings, generate a new stream header
> and clear everything for encoding a new signal.
> It' clear that current settings, re-creating windows, cpu-dependent
> functions, etc could be kept around.
> I was not quickly able to extract all the necessary initialization from
> "init_stream_internal_" into a new "FLAC__stream_encoder_restart" function.

I would theoretically accept a patch that allows direct access to the
functionality you seek, as long as it doesn't change the existing API.

Erik
-- 
--
Erik de Castro Lopo
http://www.mega-nerd.com/
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

[flac-dev] libFLAC optimizations request

[Gabriel Corneanu]

Hello all

I am using libFLAC in a corner application, compressing *a lot* of small
signals.
First is a general question: in our application we have signals in range
5-10 MHz, potentially 40MHz! Is there any potential problem with that?? The
mac sample rate is limited in flac, but it doesn't really seem to be a
problem.
The output is stored as blob in a sqlite database, it *never *needs to be a
valid audio file outside our application.
In my tests, the signals are compressed very well, much better than general
compression libraries like zlib, zstd, etc.

Now other small issues; I also made some tickets about them, but I thought
asking here might be better.

1. I would like to avoid saving vorbis comment, by default ~40 bytes. Right
now the only option is to modify stream_encoder.c, see
"metadata_has_vorbis_comment".

2. Speed is very important, therefore I would like to reuse an encoder
without re-initializing everything.
Ideally I would like 2 (exported) functions: "flush" and "restart".
"Flush" is self-explanatory, should properly end the encoding. I could
split myself "flush" from "finish", it looks relatively simple.
"Restart" should keep all current settings, generate a new stream header
and clear everything for encoding a new signal.
It' clear that current settings, re-creating windows, cpu-dependent
functions, etc could be kept around.
I was not quickly able to extract all the necessary initialization from
"init_stream_internal_" into a new "FLAC__stream_encoder_restart" function.

Regards,
Gabriel Corneanu
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev