Re: [flac-dev] 64-bit residuals

[Brian Willoughby]

Can you briefly explain the consequences of this unexpected residual?

Will a 24-bit crafted file be compressed in a lossy way?

Are there tests in the FLAC suite that would reveal this?

Can we add such a crafted 24-bit file to the test suite?

When would the new _ultrawide variants of existing functions be used?

... I ask because my use of FLAC is entirely 24-bit, so I appreciate the 
discovery of this potential issue.

Brian Willoughby

On Mar 24, 2022, at 09:05, Martijn van Beurden  wrote:
> Hi all,
> 
> I just filed an issue on github:
> https://github.com/xiph/flac/issues/306 Long story short: libFLAC
> currently uses 32-bit signed integers for residuals, but using certain
> 24-bit (crafted) material can overflow this. For examples and an
> explanation see the github issue.
> 
> I'm sending this e-mail because I'd like to attract some extra
> attention towards it. I'm working on patching this, but the residual
> is actually exported through the API (as part of the subframe) so I'd
> like to have some discussion on what would be the best way to fix
> this. I see the following options:
> 
> 1) Change all residual handling from 32-bit int to 64-bit int. This
> might incur a performance penalty and it might invalidate certain
> approaches now used with intrinsics. It also presents an API change
> 2) Change the current 32-bit signed integer arrays which are used for
> residual handling to a union within a struct. This union would hold
> both a 32-bit int and a 64-bit int variant of the residual, and the
> struct would include a bool to clarify which of the two is being used.
> For each function manipulating residuals (bitreader, bitwriter, lpc
> restore etc.) a new _ultrawide variant is written and at several
> places in the code a decision has to be added whether to use the
> _ultrawide variant or the existing ones. This changes the API a little
> more, but makes it able to keep the current intrinsics accelerated
> functions
> 3) Add a note to the FLAC spec that residuals should not exceed
> 32-bit, and add a mechanism to the encoder to ascertain this. In case
> the encoder finds a residual exceeding the 32-bit range, it defaults
> to using a verbatim subframe.
> 
> Any thoughts?

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Two questions

[Brian Willoughby]

On Aug 9, 2021, at 11:31, Martijn van Beurden  wrote:
> Op zo 8 aug. 2021 06:24 schreef Federico Miyara :
>> As I wanted to stop at sample Nend, I used Nend + 1, instead, to ensure that 
>> the sample Nend is the last one.
> 
> Considering Nend, did the count start at 0 or at 1? The flac utility starts 
> counting at 0 AFAIK, if your count starts at 1, that would explain this 
> behaviour.

Exactly. If you wanted to break a FLAC into 588-sample pieces (say, for CD 
audio blocks), then you would ask for --until=588 and you could get exactly 588 
samples numbered 0 through 587. The second piece would be another 588 samples 
numbered from 588 to 1,175.

It's always worked as I expected, but then again I'm a C programmer, not a 
Pascal programmer. ;-)

Brian

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Two questions

[Brian Willoughby]

On Aug 4, 2021, at 00:10, Federico Miyara  wrote:
> Brian,
> 
> Once more, thanks for taking your time to answer my questions and provide 
> interestig insights. Some comments below.
>  
>> I recommend writing your own utility based on the FLAC library, in C, with 
>> the features you want. I do not recall any feature in the flac command line 
>> utility that would allow this. Your workaround is a reasonable attempt, but 
>> it seems to have too many undefined side-effects.
> 
> I'm not a C programmer, which I regret, but that's how things are.

Not to wax too philosophical, but nobody is born a C programmer. Things do not 
have to remain how they are!

However, Martijn has a great suggestion: Use the metaflac command line, and I 
think this will do a lot for you.


> My quest has to do with economy, I have already a function that can read the 
> metadata from a wav file, being the only required information where the 
> metadata start (which for the wav file I know). Exactly the same function can 
> be used with a flac file without using any codec, as far as I can find out 
> the beginning of the "local copy" of the metadata.

Be careful. Many audio programs have failed by attempting to hard-code the 
beginning a certain pieces of data in a file. A related failure is to assume 
the location of the end of data. Both failures can create audio glitches.

RIFF (WAV) and IFF/FORM (AIFF) files are made from a sequence of chunks, and 
most chunks can appear in any order. I recall that WAV is slightly more 
restrictive than AIFF, but I've certainly seen errors in WAV software due to 
fixed file offsets.

Similarly, FLAC files are a sequence of blocks. Your best bet is to create 
functions that can properly scan blocks, and then once the 'riff' block is 
found, you can use a WAV function to properly scan chunks.

But don't worry too much about economy. Your function only needs to read 4 
bytes from each FLAC block in order to determine type and size. When you find 
an APPLICATION block, you can read 4 bytes to look for 'riff'. If an 
uninteresting block is found, then seek in the file according to the size of 
the block, and read 4 bytes to examine the next block. Proper code can scan 
even very large FLAC files very quickly by reading only 4 or 8 bytes at a time 
and then seeking over the uninteresting data rather than actually reading it 
from the file into memory.

When writing C code, I just use the virtual memory support to map the entire 
audio file into memory, and then only read a few bytes. The virtual memory 
paging system will not read from disk except as necessary.


>> The --keep-foreign-metadata feature was added to the command-line 
>> application after the FLAC format was finalized. The metadata ends up in an 
>> APPLICATION block, which is usually skipped by the FLAC library decoder. 
>> These are intended for third-party applications, and thus it's typically 
>> impossible to document them. Normally, a third-party software developer 
>> would add their own proprietary block to the FLAC file, and all other 
>> applications would just skip over it (because all blocks have a universal 
>> name and length at the start).
> 
> This information is most useful for me, since at least now I know the name of 
> the block containing the foreign metadata, and I know it is previous to the 
> audio data.

I never noticed that the audio block is last in a FLAC file. I'm used to AIFF 
and WAV, where chunks can almost appear in any order. This is good news because 
you'll never seek into the audio data.


> I could manually "read" the first few metadata blocks (following the format 
> specification) and found that there is a seek table whose length is roughly 
> proportional to the size of the audio samples, then a Vorbis comment 
> indicating the version of the FLAC libraru'y, and then the Application block 
> which contains the data I'm interested in. This makes its position 
> predictable so I can find it without having y¿to read all the file in search 
> of some key words!

The position is not guaranteed to be predictable. If you design your Scilab 
algorithm based on predictable positions, then you'll probably end up with 
issues. It will be more successful to create a block scanner, reading 4 or 8 
bytes per block, in order to find the APPLICATION 'riff' block that you want. 
The algorithm will be very similar to scanning chunks in the WAV file.


>> The only documentation of the APPLICATION block format is probably the 
>> source code for the flac command line utility. I did not design this, but I 
>> remember suggesting it a few times. Basically, the entire WAV or AIFF 
>> contents are in the block, verbatim, except for the chunk that would contain 
>> the audio. Since the FLAC data outside the APPLICATION block already 
>> contains the audio, that chunk is empty in the APPLICATION block.
> 
> I wonder why there is a long run of zeros (about 8192 zero bytes) in the 
> example I'm attaching, almost as long 

Re: [flac-dev] Two questions

[Brian Willoughby]

ng metadata in the audio samples!)

> However, I made a more careful test and with skip=0 until=1 and get 2 samples 
> instead of 3. 

I have used these options for very long recordings where the FLAC file was up 
to 4.0 GB in size, but the output WAV or AIFF would exceed 4.0 GB. I had to use 
--until and --skip to chop up the recording and then piece them together later 
in the DAW. I never had trouble with repeated or missing samples, but perhaps 
there are bugs related to asking for exactly zero or one sample.

I'm usually paranoid about missing or duplicating samples, so I don't think 
there's a bug with very large values for skip or until. I'm fairly certain that 
I depended on documentation such that until would not include the sample, but 
skip would. i.e. I probably calculated a split point and used something like 
the following:

flac -d --force-aiff --until=N foo.flac
mv foo.aiff foo1.aiff
flac -d --force-aiff --skip=N foo.flac
mv foo.aiff foo2.aiff

... and then used Logic Studio Pro to splice the two files together again.

If you want to contribute to FLAC, you might try a few variations with skip != 
0 and until much greater than 1. Then you can check the size of multiple test 
cases and see if there is a pattern. Maybe it only breaks with really small 
numbers, or when one value is zero.

If you find a pattern that indicates a bug, even just with the documentation, 
then please report it. Maybe you can fix it, or maybe someone else can.

Brian Willoughby

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Two questions

[Brian Willoughby]

On Aug 2, 2021, at 08:40, Federico Miyara  wrote:
> 1) Is there a way to get the audio size (number of samples) and other 
> information, such as number of channels, from a flac file without fully 
> decoding it?

Yes. There is a specific header with this information. Look for the 
documentation of the FLAC format for reference, and then look through the 
library for calls that might return this information.

I usually start from https://xiph.org/flac/documentation_format_overview.html 
for the official specification of the file format.

Note that FLAC is mostly a streaming format, where metadata is not generally 
used. But the file does have this information early on. See 
https://xiph.org/flac/format.html#def_STREAMINFO where it says that the 
STREAMINFO packet must be the first packet in the file.

> I've found that the WAV header is replicated after the "riffRIFF" keyword, 
> but I don't seem to be able to predict where it is located or whether it is 
> safe or not to asume that the first time such keyword appears is the correct 
> one, and if there is an upper bound for its location; for instance, some text 
> such as the name of a song or some comment could contain that keyword, even 
> if it is unlikely. The information preceding the  "reference libFLAC 1.3.3" 
> encoder version seems to be non-text information.

You should not use the RIFF information. It is not part of the FLAC 
specification. It is an optional enhancement to store information that is only 
about the input WAV files. It will strictly be missing from FLAC files that are 
recorded live, converted from AIFF, and will even be missing when WAV is the 
source if the option is not added.


> 2) I decode using the option --skip=0 --until=1. I would expect to get a wav 
> file with only 1 sample, but I get 3 samples. Stictly no sample should be 
> decoded since 0 samples should be skipped and the help says: 
> 
> --until={#|[+|-]mm:ss.ss}  Stop at the given sample number for each input 
> file. The given sample number is not included in the decoded output.
> 
> so sample #1 should not be included, unless the first sample is sample #0, in 
> which case only 1 sample, and not 3 samples, should be included in the output 
> file.

Are you seeing 3 bytes for 1 sample? ... or are you seeing 3 samples?

Also, I recall that the FLAC library returns 32-bit numbers, so you have to 
convert these to 16-bit or 24-bit samples.

Brian

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Autocorrelation precision insufficient

[Brian Willoughby]

Thanks.


I should probably start a new subject, but I have related questions. Please 
don't feel obligated to answer ... I'm just curious.

1) What would affect the ability of an embedded FLAC firmware to decode certain 
files?

Is it possible that the SD702 is not capable of processing complex predictors 
of higher orders?

Could it be as simple as absence (or requirement?) of mid-side encoding? (this 
doesn't seem that likely)

2) Is there a strict definition of the 9 levels of FLAC coding such that the 
level would precisely determine the predictor?

I haven't looked at the source in ages, but it seems like you're saying that 
the compression level might map to exact choices for model/predictor.

3) Can you explain why the encoded file should be exactly the same across 
different processors or implementations? I'm still fuzzy on the details, but it 
seems like the exact bits could still vary even with the same model - i.e. 
"exhaustive" implies that different implementations might stop at different 
points along the way.

Brian


On Jun 27, 2021, at 23:34, Martijn van Beurden  wrote:
> different residual encoding method in version 1.2.0, in 2007. No other
> changes to the format and thus decoding were introduced after 1.0.
> 
> The different compression levels increase one of two things: how much
> effort to spend finding the best model, and how complex a predictor to use.
> The first, the effort finding the best predictor, slows down the
> encoding but not
> decoding. These are options like adding apodizations and exhaustive model 
> order
> and precision search. With these settings, the encoder tries various models 
> and
> picks the one that fits best. The second, how complex of a predictor
> to use, does
> affect decoding performance. There are only three parameters here:
> whether to use
> a fixed or non-fixed predictor, how large the maximum model order is and 
> whether
> mid-side encoding is used.
> 
> The change I propose is a change in the way the encoder does the modelling.
> When this modelling is done with more precision, this slows down encoding,
> but it does not change anything in the way decoding is done. It just changes 
> the
> 'representation' that needs to be decoded, but this decoding process does not
> become more complex.

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Autocorrelation precision insufficient

[Brian Willoughby]

Playing Devil's Advocate here, but isn't the point of compression standards 
like FLAC that encoders can vary quite a bit, so long as decompression always 
produces the correct results? In other words, I think it might be taking it too 
far to require that the encoder produce identical results on every processor, 
at least if the decompressed audio is correct.

To counter myself on this topic, I do think it's important to maintain any 
distinction between levels (and I'm not sure how that's defined). I have the 
Sound Device 702 recorder, and it can only play back FLAC files with low to 
moderate compression. The recorder itself can only compress moderately - 
certainly nowhere near --best -9

I'd say that it's important to maintain compatibility with players that have 
limited capabilities. I don't actually know which levels of compression the 
SD702 supports, and which levels are too extreme, but whatever the threshold 
is, then I would expect that future versions of FLAC would not stop being 
compatible with the SD702 when the same compression level is used.

My apologies that I do not have more details on player limitations, but my 
point is that compatibility with players should be required, while producing 
identical compressed bit streams might not necessarily be required.

I've also never understood the mathematics behind a compression format where 
the encoder can vary immensely in quality - even to the point of exceeding 
original compression ratios - but the decoder remains compatible. Witness MP3, 
where they improved the algorithm of the encoder and produced smaller files, 
all without requiring an update to the decoder. I'm assuming that FLAC fits 
that same model.

Brian Willoughby


On Jun 25, 2021, at 14:15, Robert Kausch  wrote:
> I concur with David that compression should work the same on all platforms. 
> There can be SIMD optimizations for some routines, but they should produce 
> the same result as the non-SIMD versions. If outdated optimizations cannot be 
> updated, better drop them instead of keeping versions that produce different 
> results than the new reference code.
> 
> Regarding VSX, you could try to contact Anton Blanchard to ask whether he can 
> update the routines. Otherwise, it should be possible to emulate VSX with 
> QEMU. You won't be able to verify performance that way, of course, but should 
> be able to verify correctness.
> 
> Best,
> Robert
> 
> Am 25.06.2021 um 19:55 schrieb David Willmore:
>> Martijn,
>> 
>> Just a lone voice here, but I would opt for the best compression on the most 
>> platforms.  As I am one of the people who always run FLAC at the placebo max 
>> compression, my vote will always be for best compression over any other 
>> resource use.
>> 
>> I also can make ARMv7 hardware available to you or I can do any testing you 
>> want on a number of machines.
>> 
>> Thank you for your work!
>> 
>> Cheers,
>> David
>> 
>> On Fri, Jun 25, 2021 at 3:48 AM Martijn van Beurden  wrote:
>>>Op do 24 jun. 2021 om 09:17 schreef Martijn van Beurden
>>>:
>>>> - Switch to autoc[] as doubles, but remove current SSE and
>>>disable VSX
>>>> intrinsics for someone to update them later (I don't have any POWER8
>>>> or POWER9 hardware to test). This means all platforms will get the
>>>> same compression, but some (with only SSE but not SSE2 or with VSX)
>>>> will see a large slowdown.
>>> 
>>>I see now that besides routines with SSE intrinsics (which I rewrote
>>>into SSE2) and with VSX intrinsics (which I don't have hardware for)
>>>there is also a open pull request for routines with ARM intrinsics. I
>>>am willing and able to rewrite those if this change is accepted and
>>>merged. I have access to ARMv8 with 32-bit OS, ARMv8 with 64-bit OS,
>>>ARMv6 and I might be able to get hold of ARMv7 hardware.
> 
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] FLAC specification clarification

[Brian Willoughby]

I am also philosophically opposed to changing the specification.

That said, there's nothing wrong with adding a note to the specification about 
the common implementations, particularly the reference library. Then, future 
developers will know both the precise specification and still have the warning 
that they risk incompatibility by deviating from the reference implementation.

I own devices with FLAC implemented in firmware that is quite different from 
the reference implementation. I wouldn't want to edit the specification to 
narrow it to fit those devices, either.

Brian Willoughby

p.s. I seem to recall using a variable to shift in C, rather than a constant, 
and negative values were accepted. Then again, C started out a little weak with 
regard to certain operations, such as whether shift was signed or unsigned, 
etcetera, and thus my memory might be of a really old, esoteric implementation 
of C.


On Jun 25, 2020, at 5:09 AM, Stephen F. Booth  wrote:
> To me the real question is not whether that portion of the spec has been 
> implemented by any existing encoders/decoders but whether the spec is broken 
> (i.e. cannot be implemented as written). I don't know the rationale for 
> making the LPC shift explicitly signed. In C a negative shift is undefined 
> and it does seem in FLAC__lpc_restore_signal() for example that the LPC shift 
> is used as the argument to a right shift operation. It's possible 
> (generally/conceptually, not necessarily here) a negative shift value could 
> be used to represent a left shift. However, I know very little about linear 
> prediction and how coefficients are chosen and whether that makes sense. If 
> it really is a design flaw in the spec then it makes sense to change it or 
> document that negative shifts are not supported by any known implementation 
> as you suggest.
> 
> Philosophically my objection to changing the spec based on lack of known 
> implementations is that it could be artificially limiting, for example the 
> same argument about breaking existing decoders could be made for 32-bit 
> samples sizes.
> 
> 
> On Mon, Jun 22, 2020 at 1:33 AM Martijn van Beurden  wrote:
>> Yes, this is such a case. However, implementing this in a future 
>> encoder/decoder would break compatibility with most (likely all) existing 
>> decoders, and only in some very, very rare cases where the material is such 
>> that the encoder chooses to use negative shifts, which makes it even harder 
>> to troubleshoot. Furthermore, as this can only be used in very rare cases, 
>> there is no benefit from allowing this.
>> 
>> Op vr 19 jun. 2020 om 18:03 schreef Stephen F. Booth :
>>> Is this a case where something allowed by the specification isn't 
>>> implemented by the reference encoder/decoder (such as 25-32 bits per 
>>> sample) but could be in a different implementation? If so, I am not sure 
>>> whether it makes sense to change the specification based on the reference 
>>> implementation.
>>> 
>>> Stephen
>>> 
>>> On Wed, Jun 17, 2020 at 3:22 PM Martijn van Beurden  
>>> wrote:
>>>> Hi all,
>>>> 
>>>> When trying to better understand the way LPC exactly works, I stumbled 
>>>> upon something which, after some digging, was already reported and 
>>>> (partly) fixed: https://sourceforge.net/p/flac/bugs/424/
>>>> 
>>>> Apparently, the FLAC specification has a LPC shift that can be both 
>>>> positive and negative, but the encoder specifically makes sure that only 
>>>> positive shifts are encoded and the decoder only accepts positive shifts. 
>>>> The ffmpeg FLAC encoder and decoder show the same behaviour.
>>>> 
>>>> Now, the documentation in the source code is fixed, the documentation on 
>>>> the website (which I was looking at) isn't yet. The website format.html 
>>>> states: "Quantized linear predictor coefficient shift needed in bits 
>>>> (NOTE: this number is signed two's-complement)." The source code 
>>>> format.html says "Quantized linear predictor coefficient shift needed in 
>>>> bits (NOTE: this number is signed two's-complement; but, due to 
>>>> implementation details, must be non-negative)."
>>>> 
>>>> I was thinking of submitting a patch to the FLAC website git to get the 
>>>> website format.html up-to-date (there have been more changes than just 
>>>> this one), but I feel the line above isn't clear enough. Maybe change it 
>>>> to something like this, to make the wording more similar to the rest of 
>>>> the spec

Re: [flac-dev] identical audio but not identical unrecognized chunks

[Brian Willoughby]

On Apr 29, 2020, at 4:46 PM, Federico Miyara  wrote:
> Dear all,
> 
> I've converted a wav file to flac but during the process three wrnings were 
> casted. One of them I recall, it said that a BEXT chunk is unrecognized and 
> thus ignored.
> 
> Wouldn't it be posible to keep the unrecognized information as is when 
> decoding the file?
> 
> In general the extra chunks are located before or after the audio content, 
> and represents a tiny part of the whole file, so it oould affect only very 
> slightly the compression ratio attained.
> 
> Regards,
> 
> Federico Miyara

Federico,

flac compresses an audio file by using characteristics of the audio waveform. 
The algorithm cannot compress non-audio data. You probably already knew this. I 
just wanted to point out that flac completely ignores the non-audio data, so it 
does not even factor in to the operation, whether there are warnings, BEXT 
chunks, or not.

However, I suggested to the team many years ago that FLAC APPLICATION metadata 
blocks, which are intended for third-party applications, could be used for 
specific audio file formats such as AIFF and WAVE in order to preserve the 
non-audio chunks. As you mentioned, these extra chunks represent a small 
fraction of the total size, so adding them does not have much affect on the 
final file size.

I am happy that the suggestion was taken seriously and implemented. You will 
find that the flac command line supports an option `--keep-foreign-metadata` 
which does exactly what you seem to be suggesting, and it has been part of flac 
for many years. One of the things we discussed was the fact that some chunks 
occur before the audio chunk, and some occur after, so preserving the order 
might have been difficult. I forget the details of the discussions, but I 
believe that we eventually decided on a technique that also preserves the order 
of non-audio chunks relative to the audio chunk. This feature treats AIFF 
differently from WAVE, because it's not really possible to distinguish audio 
from non-audio without knowing what format the file is in.

A minor caveat is that you'll see a warning when using 
`--keep-foreign-metadata` - because this is a relatively new feature, the 
original designer of flac decided to let users know that it might not work 
perfectly. In my experience, it has always worked.

Another caveat is that you cannot combine `--delete-input-file` with 
`--keep-foreign-metadata` - the reason being that it's not necessarily safe to 
delete the original in the same command that you're compressing with a new 
feature. In these cases, I just manually delete the original file after 
compressing.

Yet another issue is that you cannot compress a WAVE file with 
--keep-foreign-metadata and then uncompress it into AIFF format while expecting 
the non-audio data to be preserved. flac cannot convert non-audio data from one 
format to another. Similarly, you cannot compress an AIFF file with 
--keep-foreign-metadata and then uncompress it into WAVE format while 
preserving non-audio data. flac is perfectly capable of translating the audio 
between AIFF and WAVE, but the metadata will get lost.

As a side-note, my mobile recording gear puts time stamps and other information 
into the BEXT chunk, and I usually want to preserve that. flac 
--keep-foreign-metadata does that perfectly.

On another note, there are applications like Logic Pro that add considerable 
amounts of non-audio data to a chunked file - typically these are graphical 
waveforms of the audio - and you probably do not want to waste the space needed 
to preserve these large chunks. My habit is to back up the original audio 
recordings with flac, using --keep-foreign-metadata, before loading them into 
Logic where the extra chunks will be added. This gives me the archive that I 
need without wasting extra space. Logic Pro will automatically add the waveform 
image to a file that's been freshly restored from a flac backup.

Brian Willoughby

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] FLAC frame boundaries and protocol

[Brian Willoughby]

Hello Dave.

Frames start with a 14-bit sync code, which is 13 “one" bits and 1 “zero" bit. 
Subframes start with a 1-bit padding of “zero." Keep in mind that FLAC is a bit 
stream, not a byte stream, so that 14-bit frame sync can happen anywhere in a 
pair of bytes. You can’t simply scan memory bytes for a frame sync, at least 
not unless you allow for 8 variations, apply bit masks, and allow for 
non-word-address-aligned matches.

Whenever I have a question about the FLAC format, I always start by typing 
http://flac.sourceforge.net/ into my browser, which redirects to 
https://xiph.org/flac/ these days.

There part you’re looking for is here:

https://xiph.org/flac/format.html#blocking

As for reference code, I recommend using the FLAC library. In other words, 
include the headers and link to the library, calling the API as it was 
designed. Don’t write your own code, that’s just asking for mistakes. I have 
not used the FLAC++ library, so I don’t know whether it has any peculiar 
issues. I’ve always used the Standard C FLAC library, and never have any 
trouble. You do have to pay attention to the sample format, particularly the 
bit justification for larger samples like 24-bit audio. I remember being 
interested in streaming, as opposed to file-based compression, but I can’t 
recall whether I ever finished that project.

Brian Willoughby


On Mar 5, 2019, at 10:36 AM, Dave Harris  wrote:
> Hello,
> 
> I've set up and have been reading through the FLAC reference implementation 
> source code on Windows and stepping through it in the debugger.   I've been 
> trying to understand how the protocol knows where the subframe and frame 
> boundaries are. Is there a good tutorial that discusses the ins and outs of 
> the flac protocol? Also, is there a piece of the reference code that shows 
> how frame and subframe boundaries are calculated?
> 
> Thanks.
> 
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] New ID registration

[Brian Willoughby]

A few comments:

1) Many embedded devices that have FLAC support only handle a subset of the 
FLAC capabilities. It seems highly unlikely that an embedded device would have 
the ability to parse Vorbis file format wrappers around FLAC data. In other 
words, depending upon how broad you want the support to be, it might be best to 
make this addition to FLAC rather than Vorbis.

2) In general, changes to FLAC are submitted as a PATCH. I do not believe 
anyone will add the new ID for you. You will need to add it yourself, without 
making any mistakes that would affect the existing FLAC functionality, and then 
submit your patch for approval. I am not certain that this is the correct 
approach, but it seems to be what I have witnessed. Discussion of the intended 
addition is wise, of course, as you have done.

3) I am concerned that you say this Replaygain data is large, and that it is 
float. These choices seem like overkill for a format that is lossless, and 
which basically uses fixed point integers up to 24 bits in size. Can you 
explain why the data is so large?

4) I assume that most FLAC players will ignore this App ID addition. In other 
words, I assume that only your custom player will be able to apply Replaygain. 
As a FLAC user, I would want to know that you’ve tested existing FLAC software 
and firmware to be sure that adding this App ID will not cause problems with 
existing players, especially embedded devices (e.g. Sound Devices, OPPO, etc.). 
Will Sony be performing any such tests?

Brian Willoughby


On Dec 11, 2018, at 3:05 AM, Kurosawa, Taku  wrote:
> Hi Martjn, and everyone,
>  
> Apologies if I have missed the reply, but I think I have not got any comment 
> so far on this. That means our new ID request is accepted? What should I do 
> next to proceed??  Apparently this is my first time here, so appreciate any 
> advice assistance.
>  
> Best regards,
>  
> Taku
>  
> From: Kurosawa, Taku 
> Sent: Thursday, November 22, 2018 8:41 PM
> To: Martijn van Beurden
> Cc: flac-dev@xiph.org
>  
> Hi Martijn,
>  
> Sorry for the late reply again,
> The application we are preparing this time is not exactly similar to 
> Replaygain.
>  
> Replaygain as we understand is something which normalize the loudness at 
> content provider side, but our application takes different approach. It is 
> designed to normalize the loudness at player side, according to the metadata 
> embedded within the content, which could be relatively big in its size, so it 
> will be unfit for Vorbis comment. Besides that, its concept, format are 
> incompatible with Replaygain, and we are putting effort to add additional 
> features to that in the future, which makes metadata size even bigger.
> The Player compatible with will only be the one we are developing, for now.
>  
> Best regards,
>  
> Taku
>  
> From: Martijn van Beurden 
> Sent: Friday, November 16, 2018 8:03 PM
> To: Kurosawa, Taku
> Cc: flac-dev@xiph.org
>  
> Hi,
>  
> I'm sorry, but that link doesn't seem to include anything specific about the 
> metadata itself. The reason I'm asking is because, for player compatibility, 
> it is probably much easier to use a vorbiscomment to store metadata. 
> Replaygain normalisation uses Vorbiscomment too. Most FLAC players already 
> read vorbiscomment, so for implementing plugins, this is probably much easier.
>  
> Kind regards, Martijn van Beurden
>  
> Op wo 14 nov. 2018 om 07:56 schreef Kurosawa, Taku 
> :
>> Hi, Martijn,
>>  
>> Thank you for reaching out, and apologies for late reply
>> It is a dedicated metadata for volume normalization on our player.
>> Please kindly refer to our company information here: https://labelgate.com/
>>  
>> application ID: SONN
>> application name: Sony Normalizer
>> contact e-mail address: taku.kuros...@labelgate.com
>>  
>> Taku
>>  
>> From: Martijn van Beurden
>> Sent: Wednesday, November 7, 2018 3:19 PM
>> To: flac-dev@xiph.org
>>  
>> Hi,
>> 
>> Could you please describe your application? While it is possible to use FLAC 
>> Application metadata blocks, it isn't common practice.
>> 
>> Kind regards, Martijn van Beurden
>> 
>> Op 07-11-18 om 03:14 schreef Kurosawa, Taku:
>>> To whom it may concern,
>>>  
>>> Hi, I would like to register a new ID hrere,
>>>  
>>> application ID: SONN
>>> application name: Sony Normalizer
>>> contact e-mail address: taku.kuros...@labelgate.com
>>>  
>>> please advise what the next step would be.
>>>  
>>> Best regards,
>>>  
>>> Taku Kurosawa
>> 
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] [PATCH 0/7] PowerPC64 performance improvements

[Brian Willoughby]

On Jul 11, 2018, at 5:59 PM, Anton Blanchard  wrote:
> Hi Brian,
> 
>> Thank you for this collection of patches.
>> 
>> How can I test them? What platforms (computers) have the ppc64 or
>> POWER9 processor?
> 
> The IBM Bounty Source page has a list of resources:
> 
> https://www.bountysource.com/teams/ibm/bounties
> 
> Travis also has ppc64le support, so if you add the linux-ppc64le target
> then FLAC will be tested on ppc64le.
> 
> Thanks,
> Anton

Are those all Virtual Machines? If so, how do you really know that the 
performance is 3.3x?

Brian

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] [PATCH 0/7] PowerPC64 performance improvements

[Brian Willoughby]

Thank you for this collection of patches.

How can I test them? What platforms (computers) have the ppc64 or POWER9 
processor?

Brian


On Jul 10, 2018, at 2:31 PM, Anton Blanchard  wrote:
> 
> The following series adds initial vector support for PowerPC64.
> On POWER9, flac --best is about 3.3x faster.
> 
> Amitay Isaacs (2):
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] [bug] --keep-foreign-metadata discards WAV cue markers

[Brian Willoughby]

Try using the following to decompress, instead:

flac -d —keep-foreign-metadata testmarkers.wav

Thanks for providing a test file. I was able to reproduce your issue following 
your commands, then I was able to restore the original WAV file with markers by 
adding the extra parameter. I’ve always used it this way, but often wondered 
whether the extra option was necessary. Apparently, it is.

Hope this helps,

Brian Willoughby


On May 11, 2018, at 5:21 PM, Nouvelle Collection <nouvellecollect...@gmail.com> 
wrote:
> Hello,
> I noticed that option --keep-foreign-metadata discards WAV cue
> markers. Here is how to reproduce the bug:
> 1) Create a 24-bit 96khz in SoundForge8, add 20 seconds of silence,
> and add two markers with "m" key shortcut
> 2) Save it, compress it with "flac --keep-foreign-metadata testmarkers.wav"
> 3) Decompress it with "flac -d testmarkers.flac"
> 4) Open the decompressed WAV in SoundForge: the markers have disappeared.
> 
> Here is the WAV containing markers:  http://gget.it/q4uh4/testmarkers.zip
> 
> Best regards,
> NC

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] libFLAC optimizations request

[Brian Willoughby]

I wasn’t suggesting that you run metaflac, but that you examine its source to 
see how it creates new FLAC files without the Vorbis comment. As far as I know, 
metaflac uses the standard libFLAC and creates files without the Vorbis 
overhead.

My quick review of the source seemed to indicate that calling 
FLAC__metadata_object_new(FLAC__METADATA_TYPE_VORBIS_COMMENT) will create the 
comment, but I assumed that you could pass in NULL. I mistakenly thought that 
was FLAC__stream_encoder_new().

There is a comment saying that the Ogg FLAC mapping requires a VORBIS_COMMENT, 
but that doesn’t necessarily mean that standard FLAC also requires the Vorbis 
comment.

Looking around metaflac sources, I see 
FLAC__metadata_iterator_delete_block(iterator, options->use_padding), which 
might be the code responsible for removing the Vorbis comment.

In other words, perhaps there is no existing way to avoid having the encoder 
create the Vorbis comment metadata, but there might be a way to delete that 
particular metadata before the file is saved. A closer look at how metaflac 
creates a file without Vorbis comments should help.

Brian


On Feb 4, 2018, at 6:23 AM, Gabriel Corneanu <gabrielcorne...@gmail.com> wrote:
> The problem is really as I wrote:
> 1. Metaflac is no option for me, I use libFLAC.dll
> 2. There is no way (at least how I read the code) to avoid saving
> comment with libFLAC; I would appreciate an extra option to avoid it,
> which can default to old behavior if compatibility is important.
> 3. I have a high speed application, where re-initializing an encoder
> is really significant. On corner cases it causes an 25% overhead! Of
> course I don't expect it to be that significant in normal cases.
> 
> Thanks for all replies, but I don't have the code at home.
> I will create a patch with my changes for review.
> 
> Regards,
> Gabriel
> 
> On Sun, Feb 4, 2018 at 9:57 AM, Brian Willoughby <bri...@audiobanshee.com> 
> wrote:
>> Correction, the flac command-line does create a 40-byte Vorbis comment by 
>> default. I just never noticed it before. I’ve been using —no-padding all 
>> these years for minimal file sizes without realizing that I could save 
>> another 40 bytes.
>> 
>> Anyway, since metaflac can remove the Vorbis comment using the standard 
>> library, then you should be able to create a solution without modifying 
>> libFLAC.
>> 
>> On Feb 4, 2018, at 12:43 AM, Brian Willoughby <bri...@audiobanshee.com> 
>> wrote:
>>> Gabriel,
>>> 
>>> metadata_has_vorbis_comment is a FLAC__bool which defaults to false. There 
>>> should be no reason to modify stream_encoder.c, but just modify the caller.
>>> 
>>> The following command:
>>> 
>>> metaflac —remove —block-type=VORBIS_COMMENT —don’t-use-padding
>>> 
>>> … will remove Vorbis comments from existing files, so is must be legal 
>>> without modifying the library. metaflac can clearly create a new FLAC file 
>>> without the Vorbis comment.
>>> 
>>> I use the flac command-line, and I never get Vorbis comments in the files 
>>> that I create. Perhaps you are using another tool which assumes Vorbis 
>>> comments are needed.
>>> 
>>> 
>>> The FLAC algorithm is not dependent upon sample rate. AIFF has an 80-bit 
>>> floating point type for sample rate, so it should be able to handle 40 MHz. 
>>> I assume that any AIFF can be converted to FLAC losslessly, but I have not 
>>> tested whether certain sample rates are rejected. FLAC itself only supports 
>>> sample rates up to 655,350 Hz, so you may have a problem there unless you 
>>> “lie” about the sample rate when creating the file. Maybe you could just 
>>> establish a private convention to divide the sample rate by 100 to make 
>>> yours fit. 40 MHz would map to 400 kHz, 10 MHz would map to 100 kHz, and 5 
>>> MHz would map to 50 kHz.
>>> 
>>> 
>>> You’re probably asking for trouble if you try to reuse an encoder. It seems 
>>> like there would always be some risk that details from the previous file 
>>> would bleed through into the next. Have you benchmarked allocation and 
>>> initialization? Is it really that slow? In order to reuse an encoder, 
>>> you’ll need to overwrite all state variables, and I don’t see how that 
>>> could be much faster than simply allocating them anew. Perhaps you could 
>>> allocate groups of encoders at once, if that would speed the process.
>>> 
>>> 
>>> On Feb 1, 2018, at 4:29 AM, Gabriel Corneanu <gabrielcorne...@gmail.com> 
>>> wrote:
>>>> Hello all
>>>> 
&g

Re: [flac-dev] libFLAC optimizations request

[Brian Willoughby]

Correction, the flac command-line does create a 40-byte Vorbis comment by 
default. I just never noticed it before. I’ve been using —no-padding all these 
years for minimal file sizes without realizing that I could save another 40 
bytes.

Anyway, since metaflac can remove the Vorbis comment using the standard 
library, then you should be able to create a solution without modifying libFLAC.

Brian


On Feb 4, 2018, at 12:43 AM, Brian Willoughby <bri...@audiobanshee.com> wrote:
> Gabriel,
> 
> metadata_has_vorbis_comment is a FLAC__bool which defaults to false. There 
> should be no reason to modify stream_encoder.c, but just modify the caller.
> 
> The following command:
> 
> metaflac —remove —block-type=VORBIS_COMMENT —don’t-use-padding
> 
> … will remove Vorbis comments from existing files, so is must be legal 
> without modifying the library. metaflac can clearly create a new FLAC file 
> without the Vorbis comment.
> 
> I use the flac command-line, and I never get Vorbis comments in the files 
> that I create. Perhaps you are using another tool which assumes Vorbis 
> comments are needed.
> 
> 
> The FLAC algorithm is not dependent upon sample rate. AIFF has an 80-bit 
> floating point type for sample rate, so it should be able to handle 40 MHz. I 
> assume that any AIFF can be converted to FLAC losslessly, but I have not 
> tested whether certain sample rates are rejected. FLAC itself only supports 
> sample rates up to 655,350 Hz, so you may have a problem there unless you 
> “lie” about the sample rate when creating the file. Maybe you could just 
> establish a private convention to divide the sample rate by 100 to make yours 
> fit. 40 MHz would map to 400 kHz, 10 MHz would map to 100 kHz, and 5 MHz 
> would map to 50 kHz.
> 
> 
> You’re probably asking for trouble if you try to reuse an encoder. It seems 
> like there would always be some risk that details from the previous file 
> would bleed through into the next. Have you benchmarked allocation and 
> initialization? Is it really that slow? In order to reuse an encoder, you’ll 
> need to overwrite all state variables, and I don’t see how that could be much 
> faster than simply allocating them anew. Perhaps you could allocate groups of 
> encoders at once, if that would speed the process.
> 
> 
> On Feb 1, 2018, at 4:29 AM, Gabriel Corneanu <gabrielcorne...@gmail.com> 
> wrote:
>> Hello all
>> 
>> I am using libFLAC in a corner application, compressing a lot of small 
>> signals.
>> First is a general question: in our application we have signals in range 
>> 5-10 MHz, potentially 40MHz! Is there any potential problem with that?? The 
>> mac sample rate is limited in flac, but it doesn't really seem to be a 
>> problem.
>> The output is stored as blob in a sqlite database, it never needs to be a 
>> valid audio file outside our application.
>> In my tests, the signals are compressed very well, much better than general 
>> compression libraries like zlib, zstd, etc.
>> 
>> Now other small issues; I also made some tickets about them, but I thought 
>> asking here might be better.
>> 
>> 1. I would like to avoid saving vorbis comment, by default ~40 bytes. Right 
>> now the only option is to modify stream_encoder.c, see 
>> "metadata_has_vorbis_comment".
>> 
>> 2. Speed is very important, therefore I would like to reuse an encoder 
>> without re-initializing everything.
>> Ideally I would like 2 (exported) functions: "flush" and "restart".
>> "Flush" is self-explanatory, should properly end the encoding. I could split 
>> myself "flush" from "finish", it looks relatively simple.
>> "Restart" should keep all current settings, generate a new stream header and 
>> clear everything for encoding a new signal.
>> It' clear that current settings, re-creating windows, cpu-dependent 
>> functions, etc could be kept around. 
>> I was not quickly able to extract all the necessary initialization from 
>> "init_stream_internal_" into a new "FLAC__stream_encoder_restart" function.
>> 
>> Regards,
>> Gabriel Corneanu

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] libFLAC optimizations request

[Brian Willoughby]

Gabriel,

metadata_has_vorbis_comment is a FLAC__bool which defaults to false. There 
should be no reason to modify stream_encoder.c, but just modify the caller.

The following command:

metaflac —remove —block-type=VORBIS_COMMENT —don’t-use-padding

… will remove Vorbis comments from existing files, so is must be legal without 
modifying the library. metaflac can clearly create a new FLAC file without the 
Vorbis comment.

I use the flac command-line, and I never get Vorbis comments in the files that 
I create. Perhaps you are using another tool which assumes Vorbis comments are 
needed.


The FLAC algorithm is not dependent upon sample rate. AIFF has an 80-bit 
floating point type for sample rate, so it should be able to handle 40 MHz. I 
assume that any AIFF can be converted to FLAC losslessly, but I have not tested 
whether certain sample rates are rejected. FLAC itself only supports sample 
rates up to 655,350 Hz, so you may have a problem there unless you “lie” about 
the sample rate when creating the file. Maybe you could just establish a 
private convention to divide the sample rate by 100 to make yours fit. 40 MHz 
would map to 400 kHz, 10 MHz would map to 100 kHz, and 5 MHz would map to 50 
kHz.


You’re probably asking for trouble if you try to reuse an encoder. It seems 
like there would always be some risk that details from the previous file would 
bleed through into the next. Have you benchmarked allocation and 
initialization? Is it really that slow? In order to reuse an encoder, you’ll 
need to overwrite all state variables, and I don’t see how that could be much 
faster than simply allocating them anew. Perhaps you could allocate groups of 
encoders at once, if that would speed the process.


On Feb 1, 2018, at 4:29 AM, Gabriel Corneanu  wrote:
> Hello all
> 
> I am using libFLAC in a corner application, compressing a lot of small 
> signals.
> First is a general question: in our application we have signals in range 5-10 
> MHz, potentially 40MHz! Is there any potential problem with that?? The mac 
> sample rate is limited in flac, but it doesn't really seem to be a problem.
> The output is stored as blob in a sqlite database, it never needs to be a 
> valid audio file outside our application.
> In my tests, the signals are compressed very well, much better than general 
> compression libraries like zlib, zstd, etc.
> 
> Now other small issues; I also made some tickets about them, but I thought 
> asking here might be better.
> 
> 1. I would like to avoid saving vorbis comment, by default ~40 bytes. Right 
> now the only option is to modify stream_encoder.c, see 
> "metadata_has_vorbis_comment".
> 
> 2. Speed is very important, therefore I would like to reuse an encoder 
> without re-initializing everything.
> Ideally I would like 2 (exported) functions: "flush" and "restart".
> "Flush" is self-explanatory, should properly end the encoding. I could split 
> myself "flush" from "finish", it looks relatively simple.
> "Restart" should keep all current settings, generate a new stream header and 
> clear everything for encoding a new signal.
> It' clear that current settings, re-creating windows, cpu-dependent 
> functions, etc could be kept around. 
> I was not quickly able to extract all the necessary initialization from 
> "init_stream_internal_" into a new "FLAC__stream_encoder_restart" function.
> 
> Regards,
> Gabriel Corneanu

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] using libflac++ on a live internet stream

[Brian Willoughby]

I don't have time to look into this, at the moment. Hopefully, someone else on 
the list can lend a hand.

It's been about ten years since I wrote any new code using libFLAC, so it would 
take me a while to help find any problems with your code. Perhaps the holidays 
will afford some time.

Brian


On Dec 13, 2017, at 8:12 AM, Chris Barrett <cbarrett.c...@gmail.com> wrote:
> Thanks Brian.  I converted everything to libFLAC and got the same results.
> 
> [snip]
> 
> I can send out some of the code fragments if needed.
> 
> Thanks,
> Chris
> 
> On Tue, Dec 12, 2017 at 4:44 PM, Brian Willoughby <bri...@audiobanshee.com> 
> wrote:
>> Hi Chris,
>> 
>> Have you tried the Standard C libflac option? Not that I have anything 
>> against the C++ flavor, but I've only ever worked with the C API, to keep 
>> things simple. Sorry for the brief response, but I wanted to reply quickly.
>> 
>> Brian
>> 
>> On Dec 12, 2017, at 1:51 PM, Chris Barrett <cbarrett.c...@gmail.com> wrote:
>> > I'm trying to use libflac++ on a live internet audio stream.  I don't see 
>> > anything mentioned in the documentation that suggests this should not be 
>> > possible, so I hope I'm not chasing down the wrong path (two weeks in).
>> >
>> > The encoder seems to be working fine and creates data for the 
>> > write_callback, which I have coded to packetize that data and send it 
>> > across the network.
>> >
>> > The question is how the decoder at the receiver should operate.  Ideally, 
>> > when a packet arrives, it can be fed to the decoder.  When the decoder has 
>> > enough data for a new uncompressed frame, it returns one.
>> >
>> > But the decoder API does not seem setup for this type of synchronous 
>> > operation.  When a packet arrives, I'm trying to call process_single_frame 
>> > and then supply the packet data in the read_callback.  It takes the data 
>> > but always calls the read_callback again for more data.  It does call 
>> > error_callback (status == 0) once, but never write_callback.  So, I'm 
>> > guessing that the packet does not have enough data for a frame.
>> >
>> > So, I tried queueing write_callbacks at the encoder before packetizing it 
>> > and sending it.  It seems like encoder init causes three write_callbacks 
>> > that make up the metadata, and then each call to process causes one 
>> > write_callback that is presumably the frame data.  The documentation 
>> > states that one cannot rely on this, but I can't see any other choice.  
>> > Just to be safe, I let the metadata and the first four frames queue up 
>> > before sending it.  Still the decoder asks for more data when I call 
>> > process_single_frame with this data and never calls write_callback.  I 
>> > don't have the decoder meta callback installed, but could try that to test 
>> > if it is at least decoding the metadata.
>> >
>> > Any help is greatly appreciated.
>> >
>> > Thanks,
>> > Chris
>> >
>> 
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] using libflac++ on a live internet stream

[Brian Willoughby]

Hi Chris,

Have you tried the Standard C libflac option? Not that I have anything against 
the C++ flavor, but I've only ever worked with the C API, to keep things 
simple. Sorry for the brief response, but I wanted to reply quickly.

Brian


On Dec 12, 2017, at 1:51 PM, Chris Barrett  wrote:
> I'm trying to use libflac++ on a live internet audio stream.  I don't see 
> anything mentioned in the documentation that suggests this should not be 
> possible, so I hope I'm not chasing down the wrong path (two weeks in).
> 
> The encoder seems to be working fine and creates data for the write_callback, 
> which I have coded to packetize that data and send it across the network.
> 
> The question is how the decoder at the receiver should operate.  Ideally, 
> when a packet arrives, it can be fed to the decoder.  When the decoder has 
> enough data for a new uncompressed frame, it returns one.
> 
> But the decoder API does not seem setup for this type of synchronous 
> operation.  When a packet arrives, I'm trying to call process_single_frame 
> and then supply the packet data in the read_callback.  It takes the data but 
> always calls the read_callback again for more data.  It does call 
> error_callback (status == 0) once, but never write_callback.  So, I'm 
> guessing that the packet does not have enough data for a frame.
> 
> So, I tried queueing write_callbacks at the encoder before packetizing it and 
> sending it.  It seems like encoder init causes three write_callbacks that 
> make up the metadata, and then each call to process causes one write_callback 
> that is presumably the frame data.  The documentation states that one cannot 
> rely on this, but I can't see any other choice.  Just to be safe, I let the 
> metadata and the first four frames queue up before sending it.  Still the 
> decoder asks for more data when I call process_single_frame with this data 
> and never calls write_callback.  I don't have the decoder meta callback 
> installed, but could try that to test if it is at least decoding the metadata.
> 
> Any help is greatly appreciated.
> 
> Thanks,
> Chris
> 
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] FLAC compression experiment

[Brian Willoughby]

There are several errors and misinterpretations in your example.

Your first error is that at 5% volume, starting with a 16-bit file results in 
less than 12 bits of resolution. There is no way to get back the original 
16-bit quality unless you promote the 16-bit file to a 24-bit file at the same 
time as you reduce the volume to 5%. While that might work at the expense of 
using 24-bit FLAC for a 16-bit WAV, it would fail to be lossless if you start 
with a true 24-bit WAV. In the latter case, your FLAC would only preserve less 
than 20 bits of audio, and that is not lossless.

Your second error is that WAV file sizes are always identical for all WAV files 
with the same bit depth and time duration (and channel count - in this case, 
stereo). That is the definition of uncompressed audio file sizes. However, even 
with identical source WAV file sizes, the corresponding FLAC file sizes are 
expected to vary a great deal because FLAC is required to be lossless. An MP3 
can have a fixed compression ratio because it is lossy - sound quality is lost. 
FLAC cannot have a fixed compression ratio because is must be lossless, and 
thus differences in the input audio will change the size. That is expected.

The misinterpretation I refer to is that the relationship between the volume 
level (amplitude) of the original audio and the size of the compressed FLAC 
file is expected to be exactly like you have discovered. FLAC uses many 
mathematical algorithms to reduce the size of audio without losing any 
information. One of these processes is to use differential values rather than 
absolute values. The second is to use Rice coding so that small values take 
less space than large values. There's a lot more going on than just those two 
algorithms, but the point is that sounds with a lower amplitude will always 
compress smaller than sounds with a higher amplitude.

I can guarantee, mathematically, that there is no way to reduce the volume of a 
16-bit audio recording to 5% and then restore is to it's original value (at 
least not without cheating and using 24-bit storage in between) without losing 
something. Your 5% volume is a reduction in signal-to-noise ratio of 26 dB, and 
there's no way to get that back losslessly.

In other words, your suggestion to add a volume header for normalization 
defeats the purpose of FLAC, because your technique will not be lossless. There 
are plenty of compression algorithms that can lose 26 dB of S/N and create 
smaller files.

Thanks for writing, though. It's always fun to discover how FLAC works. In my 
personal usage, I make live recordings and then use FLAC to archive the 
originals. Since live recording should never allow the volume to reach the Full 
Scale limits, my live recordings always compress smaller than the equivalent 
24-bit audio for distribution. I discovered the same things you discovered when 
I went through the mastering process - which increases the volume - and noticed 
that the final FLAC files were much larger than the original FLAC files. This 
was really only because I increased the volume. In this case, I might be able 
to release unmastered audio recordings with a "normalization" value and keep 
the FLAC files smaller. But that only works because the original audio is 
already quieter. It doesn't work losslessly if you artificially reduce the 
volume - because then you're losing information.

Brian Willoughby


On Aug 19, 2017, at 2:37 PM, z1x2c3v4z1x2c3v4 <z1x2c3v4z1x2c...@hotmail.com> 
wrote:
> I feel I have found a super high compression way of FLAC. I have tested a 1 
> hour WAV file of 440HTZ with a 5,25,50,75,100 normalize volume preset. This 
> dramatically changes the compression size of the end FLAC file even though 
> the WAV file size is identical for all 5 WAV files. Only the volume is 
> different. When you renormalize the WAV to its original volume the file is 
> still 100% original.
> 
> My tests are the following.
> source file WAV. 1 hour 440HTZ tone. 600MB.
> volume at 100. FLAC file size is  184MB.
> volume at 75. FLAC file size is  166MB.
> volume at 50. FLAC file size is  143MB.
> volume at 25. FLAC file size is  109MB.
> volume at 5. FLAC file size is  96MB.
> 
> From184MB to 96MB.
> As you can see from the full 100 normalize volume to the 5 normalize volume 
> it is half the file size.
> So to put it. I think that you should have your compression of FLAC to also 
> include a normalize volume shift to further compress the end product. Either 
> have a default setting or allow a user to input an option in the command line 
> arguments to add a normalize volume value. Then to uncompress it back to a 
> WAV file FLAC would read the volume header in the FLAC file to apply the WAV 
> original normalize volume and expand it back to its original state. 
> 
> Then have the FLAC decoder for all the players out there to read the volume 
> head

Re: [flac-dev] [Cellar] FLAC Markdown

[Brian Willoughby]

Hi Dave,

I don't have older FLAC sources handy to check, but I believe that text like 
METADATA_BLOCK_HEADER was taken literally from the C language header files. 
These days, I believe that such types have been renamed like 
FLAC__METADATA_TYPE_HEADER to avoid potential conflicts with libraries outside 
the FLAC realm.

I don't think that the outdated labels should remain, but rather than convert 
them to human language text, I think it would be a more worthwhile effort to 
simply update these all-caps labels to match the current source code. That 
would be slightly more work than your proposal, because you'd have to search 
for the exact structure names in the latest sources, but it would be infinitely 
more useful to those who read the spec.

It is my opinion that most people who are looking into low-level details like 
headers and meta data format will all be working with one or both of the C 
language FLAC or C++ language FLAC++ libraries. I think that's the advantage 
you're looking for. The FLAC libraries have been ported to all major operating 
systems as well as many embedded platforms, so this C language is quite 
prevalent.

Brian Willoughby
Sound Consulting


On Jun 6, 2017, at 5:23 PM, Dave Rice <d...@dericed.com> wrote:
> Hi all,
> 
> On Jun 5, 2017, at 11:52 PM, Andrew James Weaver <we...@uw.edu> wrote:
>> Hello all!
>> (cc-ing the flac-dev list)
>> 
>> I would like to give an update as to the recent CELLAR work on the FLAC 
>> specification.
>> 
>> • Work has been done to make internal and external links more accurate and 
>> reliable.
>> • 'Rice Coding' has been clarified as 'Exponential Golomb Coding.'
>> • Clarifications have been made for binary representation.
>> • Typos and other small changes have been fixed for clarity.
>> 
>> Lastly, a version 00 release has been made (available at 
>> https://github.com/privatezero/flac_markdown/releases) and the draft 
>> document has been uploaded to the IETF datatracker 
>> (https://datatracker.ietf.org/doc/draft-xiph-cellar-flac/)
> 
> From a review of the markdown and the rendered RFC, I’d like to consider 
> renaming many of the components of FLAC from the 
> ALL_CAPS_WITH_UNDERSCORE_STYLE to `Title Case Names in BackTicks` (which is 
> similar to how local components have been named and referenced in the FFV1 
> and Matroska documents). Thus “METADATA_BLOCK_HEADER” would be `Metadata 
> Block Header` and expressed with that same case and quoting at each 
> reference. I also suggest that the metadata block types such as 
> “METADATA_BLOCK_STREAMINFO” and “METADATA_BLOCK_APPLICATION” simply be 
> renamed as `Streaminfo Block` and `Application Block` and that each of those 
> components be defined as being a `Metadata Block`.
> 
> I may send a pull request for this soon but wanted to ask around in case 
> there’s some reason for or advantage of the ALL_CAPS_WITH_UNDERSCORE_STYLE 
> that I’m missing.

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] warning that legacy WAVE file has format type 1 but bits-per-sample is 24

[Brian Willoughby]

On May 26, 2017, at 9:52 PM, Erik de Castro Lopo <mle...@mega-nerd.com> wrote:
> Mark H. David wrote:
>> Notice the warning was given only for the WAVE file produced by the
>> flac reference executable.
> 
> I agree. The flac decoder should not generate WAV files which it 
> warns about when encoding.
> 
>> Should I file a bug for this?
> 
> No need. This is sufficient.
> 
> Erik

Are there some (old) programs that will not accept WAVEX headers? I'm trying to 
think of why the flac utility worked this way in the first place.

In any event, it seems like it would be prudent to add a command line option to 
preserve the existing behavior, just in case anyone has archived their audio 
using FLAC and would need to maintain compatibility with the legacy WAVE 
format. It's not always possible to replace certain old programs, especially in 
recording studios.

Brian Willoughby
Sound Consulting

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] [PATCH 2/5] SIMD: remove extra space

[Brian Willoughby]

I agree with Erik regarding this patch.


On Feb 19, 2017, at 12:17 PM, Erik de Castro Lopo  wrote:
> lvqcl wrote:
>> Most libFLAC code don't have a space between if and a parenthesis,
>> so the patch removes them from lpc_intrin_sseNN.c files.
> 
> I don't really agree with this one. If anything I think we should
> slowly move to making the code insert a space between the `if` and
> the open parenthesis because its easier to read.
> 
> Erik

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Flac multi channel

[Brian Willoughby]

Don't overlook the FLAC in Ogg container solution. That's established as a 
standard for some time now, as far as I know, and would probably be better than 
a new, proprietary multi mono bundle. I haven't used it myself, but people have 
been talking about it for a while, and I believe that some "FLAC" users are 
actually working with FLAC in Ogg container files.

Brian Willoughby


On Jan 27, 2017, at 12:55 AM, Olivier Tristan <o.tris...@uvi.net> wrote:
> Thanks everybody for their answer.
> 
> This is quite unfortunate for me, but hey, that's life.
> 
> I will probably end up doing some multi mono bundle similar to what Protools 
> did back in the days with its .L .R files
> 
> Le 26/01/2017 à 18:58, Martin Leese a écrit :
>> Federico Miyara wrote:
>> ...
>>> The file format allows some unused fields for future use, such as the
>>> padding block. It could include a flag to indicate a change in the
>>> format adding one more streaminfo byte which would allow up to 256
>>> channels (actually, 256 + 8), or it could trigger a new byte when .
>>> 
>>> There is also an invalid block identifier (127) which could be used with
>>> the same purpose.
>> The problem isn't *just* the 3-bit field used for
>> the number of channels.  As Brian Willoughby
>> explained:
>> ...
>>>> As you cram more channels into a block, you get fewer samples per block for
>>>> each individual channel. There simply isn't any advantage to having lots of
>>>> channels in a single stream.
>>>> 
>>>> I believe that Ogg allows you to create a file that interleaves multiple
>>>> FLAC files.
>> Perhaps comparing FLAC with the Ogg
>> container and Vorbis codec will aid
>> understanding.
>> 
>> With Ogg, different streams can be either
>> chained (sequential) or grouped
>> (parallel/interleaved).  Typically, metadata
>> streams would be chained (so they appear
>> before any audio data) and audio streams
>> would be grouped.
>> 
>> Within a single FLAC stream the audio is
>> split into blocks which are grouped.  But within
>> each block the eight channels are chained.
>> This makes sense with a maximum of only
>> eight channels.  Within a Vorbis stream the
>> audio is split into frames which are grouped.
>> Because a Vorbis stream can contain up to
>> 256 channels, within each frame the channels
>> are also grouped.
>> 
>> So the maximum of eight channels is really
>> embedded into the FLAC standard.  To change
>> this would require a whole new standard (or
>> the use of multiple grouped FLAC streams in
>> an Ogg container).
>> 
>> Regards,
>> Martin
> 
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Flac multi channel

[Brian Willoughby]

For one thing, only stereo files benefit from additional compression by taking 
advantage of similar audio in the two channels. FLAC files with more than two 
channels are not any more efficient than separately-compressed monophonic 
files. In fact, depending upon the audio content, you're probably better off 
encoding an eight-channel set as four stereo FLAC files, even though FLAC 
supports eight-channel files already.

At a certain point, you have to put a limit on block sizes to accommodate 
embedded devices, not to mention the streaming latency of large data sets. As 
you cram more channels into a block, you get fewer samples per block for each 
individual channel. There simply isn't any advantage to having lots of channels 
in a single stream.

I believe that Ogg allows you to create a file that interleaves multiple FLAC 
files. So, if you want high channel counts then it seems like FLAC in Ogg is 
the choice (but I have not experimented with this option, personally).

Brian Willoughby
Sound Consulting


On Jan 25, 2017, at 11:27 AM, Olivier Tristan <o.tris...@uvi.net> wrote:
> I see :(
> 
> That what I would call a good struct size optimisation. 
> Please tell me there was another reason behind this being only 3 instead of 8 
> or 16 bits, right ?
> 
> 2017-01-25 18:30 GMT+01:00 Tor-Einar Jarnbjo <tor-ei...@jarnbjo.name>:
>> Hello Olivier,
>> 
>> the limitation is in the file format itself, as the number of channels is 
>> encoded in a 3 bit field in the streaminfo metadata block. 
>> 
>> Tor-Einar 
>> 
>> Am 25. Januar 2017 16:45:32 MEZ schrieb Olivier Tristan <o.tris...@uvi.net>:
>>> Hi Guys,
>>> 
>>> I know that FLAC format is currently limited to 8 channels but I was 
>>> wondering if this hard limitation of the format
>>> 
>>> or if it can be easily circumvented if the flac library is compiled with 
>>> other settings and/or the software using it don't mind it

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Do we need a pre-release?

[Brian Willoughby]

On Dec 8, 2016, at 12:59 AM, Dave Yeo <dave.r@gmail.com> wrote:
> On 12/08/16 12:24 AM, Thomas Zander wrote:
>> On 7 December 2016 at 21:08, Erik de Castro Lopo <mle...@mega-nerd.com> 
>> wrote:
>>> lvqcl.mail wrote:
>>> 
>>>> "make -f Makefile.lite" also doesn't work out of box.
>>> 
>>> Didn't work for the 1.3.1 release either. Makes me wonder why we even
>>> keep it around.
>> 
>> Because it works on many systems with only tiny tweaks (e.g. passing
>> an env var) without the mess that is GNU autotools.
>> I use it frequently and always thought of it of an fine asset to have
>> (which other projects lack).
>> Would be nice to keep it IMHO.
> 
> I agree. Here it would require a lot of tweaking but it is still a good 
> beginning point for someone who needs a plain makefile.
> Even if slightly broken, it should be kept as a template.
> Dave

Maybe add a README.lite in that directory, or a section in an existing README, 
to explain the purpose of Makefile.lite and the fact that it may not work 
exactly as provided? I'm not building for any of the systems that would use it, 
so this is just my comment from the peanut gallery.

Brian Willoughby

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] test_streams dependencies

[Brian Willoughby]

My assumption is that the code was written to call flac_fopen() so that it is 
portable to every operating system, even those without fopen(). If you replace 
flac_fopen() with fopen(), then the tests won't compile on systems that don't 
have fopen().

You could simply have the following:

#if defined _MSC_VER || defined __MINGW32__
#define flac_fopen fopen
#endif

In a header, or simply at the top of test_streams/main.c where there's already 
some conditional compilation.

Brian


On Jan 31, 2016, at 7:45 AM, lvqcl  wrote:
> test_streams currently depends on grabbag and (on Windows) on win_utf8_io 
> libs.
> It depends on win_utf8_io only because it uses flac_fopen() function.
> 
> It will become to depend on libFLAC when all file functions will be moved
> from win_utf8_io to libFLAC. Not a big problem, but it is possible to avoid
> this dependency by replacing flac_fopen() with fopen().
> 
> test_streams doesn't open/create files with unicode filenames, so it won't
> affect its functionality.
> 
> So, what's better:
> a) replace all flac_fopen() calls inside test_streams/main.c with fopen()
> b) leave it as is, so that on Windows build of test_streams will depend on 
> libFLAC
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] question about src/test_seeking.c - seek_barrage()

[Brian Willoughby]

Agreed.

Although there wasn't a universal standard when FLAC was started, it sure seems 
like int64_t and related types are available on all systems these days. Either 
FLAC__intXX or intXX_t are better than the old types, like byte, word, dword, 
quadword, or whatever.

Brian


On Jan 31, 2016, at 4:09 PM, Erik de Castro Lopo  wrote:
> If I was writing FLAC from scratch now, I would not define any of these
> FLAC_intXX types and just use the standard  types.
> 
> Since this `n` variable is not part of the public API, I would use int64_t.
> 
> Erik
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] FLAC__BYTES_PER_WORD==8 test results

[Brian Willoughby]

On Dec 30, 2015, at 3:09 AM, Thomas Zander  
wrote:
> On 29 December 2015 at 21:50, lvqcl  wrote:
>> So, does it make sense to #define FLAC__BYTES_PER_WORD (in bitreader.c)
>> as 4 for 32-bit and as 8 for 64-bit targets?
> 
> Your tests so far imply this is a sensible default.
> I'd say go ahead. We can always change it if it turns out there is a
> better option.
> 
> Riggs

What's wrong with something incredible simple, like:

#define FLAC__BYTES_PER_WORD (sizeof(int))

?

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] How to check for 64-bit CPU?

[Brian Willoughby]

It seems that most compilers define some sort of architecture values, but 
they're not necessarily consistent across all compilers. I believe that the 
original point of the FLAC__CPU_X86_64 and similar defines was to make these 
consistent within the FLAC sources, but not to reinvent the wheel entirely.

As others have pointed out, there are autoconf solutions for architecture 
detection, and it would make sense to leverage these and set FLAC__CPU_X86_64 
and related defines in the headers based on config.

Don't forget cross-compiling. Sometimes a developer wants to build for a 
different architecture than the one running.

Brian


On Dec 28, 2015, at 11:35 AM, lvqcl  wrote:
> In stream_encoder.c there's the following code:
> 
>  #if defined FLAC__CPU_X86_64 /* and other 64-bit arch, too */
>  if(mean <= 0x8000/512) { /* 512: more or less optimal for both 16- 
> and 24-bit input */
>  #else
>  if(mean <= 0x8000/8) { /* 32-bit arch: use 32-bit math if possible */
>  #endif
> 
> A) How to properly check for 64-bit architectures?
> I can check for "defined FLAC__CPU_X86_64" or "defined _WIN64".
> Is it possible to use SIZEOF_VOIDP? such as "#if SIZEOF_VOIDP == 8" ?
> 
> B) Does it make sense to put the following code into some header file?
> (if yes, what header file should be used?)
> 
> 
>  #if (defined FLAC__CPU_X86_64) || ...something else here...
>  #define FLAC__64BIT_ARCH 1
>  #else
>  #undef FLAC__64BIT_ARCH
>  #endif
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] flac-dev Digest, Vol 131, Issue 8

[Brian Willoughby]

Trojan Horse software? Running on a list-member's computer?

I deleted the message already, but the headers might have some clues. I'm not 
too worried unless it happens again.

Brian


On Oct 24, 2015, at 12:16 PM, Marcus Johnson  wrote:
> How the hell did this spam get through?​
> 
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Supporting 32 bit data

[Brian Willoughby]

On Oct 1, 2015, at 1:22 PM, Martijn van Beurden <mva...@gmail.com> wrote:
> Op 01-10-15 om 18:14 schreef lvqcl:
>> Currently libFLAC stores residual signal as 32-bit signed int. And there
>> are the following comments in stream_encoder.c:
> 
> The residual is stored as a Golomb/Rice code. As far as I know, 
> that is not limited to 32-bit in the format itself, only in the 
> implementation.
> 
> However, there are two residual coding methods now: rice and 
> rice2. rice2 was added when it turned out rice (with a 4 bit 
> rice parameter) wasn't able to cope with the larger residuals 
> that 24-bit material. It looks like a 5-bit rice parameter is 
> enough for 32-bit material as well (highest divisor is 2^30, 
> whereas a 4-bit parameter is limited to 2^14), but maybe a rice 
> partition with 6 bits has to be added, who knows?
> 
> Anyway, I don't think the format is in any way unsupportive of 
> 32-bit int material, it's just the current implementation of the 
> codec.
I've been meaning to take a look at the header format to see whether 32-bit 
would confuse old decoders, and today's messages reminded me to do that.

METADATA_BLOCK_STREAMINFO allows "bits per sample" to range from 4 bits to 32 
bits. That means it should be much less complicated to support 32-bit encoding 
without completely changing the format.

Decoders already have to check these headers to compare what they support 
against the specifics of a given file. I don't think that error reporting is 
very good on embedded FLAC implementations - they usually silently fail to play 
if given an encoded file that they can't handle. Adding 32-bit support should 
not make this any worse than it already is.

In contrast, attempting something crazy like support for 48-bit samples would 
either completely break the format for backwards compatibility, or at the very 
least would require using some of the "reserved" field values to extend the 
headers in one way or another. I don't recommend going down this route unless 
there is an extremely compelling reason.

A minor headache is that the FRAME_HEADER does not support 32-bit sample sizes 
on its short list, but there is an option to defer to the STREAMINFO for sample 
size. That might pose a challenge for 32-bit files. Not sure yet.

Brian Willoughby
Sound Consulting

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] config.rpath?

[Brian Willoughby]

On Aug 1, 2015, at 7:50 PM, Dave Yeo dave.r@gmail.com wrote:
 On 08/01/15 12:55 AM, Erik de Castro Lopo wrote:
 I've just switched autogen.sh from bash to sh and made sure
 it works with /bin/bash
 
 Plus symlinks are unwanted.
 I've also added a --no-symlinks option to autogen.sh.
 
 Happy now?:-)
 
 Yes, always nice to have plain sh being the requirement.


Agreed.

 Also tested with [pdk]sh.
 Dave

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

[flac-dev] Apple afconvert SRC (was: FLAC implementation in Windows 10)

[Brian Willoughby]

Sorry for the non-FLAC content, but I will say that a utility like afconvert 
that also supported FLAC would be great. It would be very useful to be able to 
convert directly from a FLAC that is very large to a CAF, since the latter can 
exceed 4 GB without breaking the format.


On Jul 19, 2015, at 2:03 PM, Marcus Johnson bumblebritche...@gmail.com wrote:
 Do you have a source on the CoreAudio quality thing? I'd be very interested 
 in reading more about that.

The following thread seems to touch on all the key words.

http://www.macosxaudio.com/forums/viewtopic.php?f=30t=45889

Depending upon whether you're a developer and/or user, you might want to search 
further.

Brian

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] FLAC implementation in Windows 10

[Brian Willoughby]

On Jul 14, 2015, at 8:18 AM, Declan Kelly flac-...@groov.ie wrote:
 On Mon, Jul 13, 2015 at 01:28:22PM +0200, mva...@gmail.com wrote:
 FLAC is not the only one though, Apple Lossless has been added 
 to the mix in the same way, but (properly) creates smaller files.
 
 Can anyone on the list (possibly someone who works for MSFT) get this
 fixed before Win10 is released?

What size differences are we talking about? How many types of files are 
affected? What is the percent difference in size for the typical file?

Brian

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] [PATCH] Fix for odd RIFF size

[Brian Willoughby]

The ckSize field can be odd to represent the size of the valid data.

However, the chunk itself must always be an even size. This requires a padding 
byte at the end of a chunk before the next chunk can begin, or before the end 
of file. The latter case is the one that most often occurs in buggy RIFF 
writing programs - the last chunk will have an odd ckSize and the file will be 
one byte shorter than a valid RIFF.

Some RIFF readers blindly expect an even number of bytes in every chunk, and 
seek past each chunk to see if another chunk exists afterwards. In the case 
where the last chunk ends a byte earlier than expected, this class of RIFF 
reader gets confused and fails.


So, to reiterate:

1) odd value in ckSize is correct, when appropriate
2) odd size of chunk is not correct, not under any conditions
3) flac should read RIFF anyway, if the only problem is the final padding byte 
in the chunk

Note: if a RIFF is truncated more severely than just a missing padding byte, 
then it should be considered an error that is reported to the user. It's 
difficult to preserve lossless quality when input data is missing!

I did not review this patch, but I wanted to clarify the RIFF specification on 
this subject, and to share my experience with common RIFF writer/reader bugs.

Brian Willoughby


On Jul 9, 2015, at 2:28 PM, lvqcl lvqcl.m...@gmail.com wrote:
 This patch should fix ticket https://sourceforge.net/p/flac/bugs/419/
 and its duplicate https://sourceforge.net/p/flac/support-requests/152/
 
 some programs write odd value to ckSize of RIFF chunk. Not sure
 is it correct or not, but flac should read them anyway.odd_riff_size.patch

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] num_comments==0 and comments==0

[Brian Willoughby]

What is the advantage of removing an assert? - even FLAC_ASSERT()

My understanding is that assert() is only compiled into the code with Debug 
builds, whereas with a Release build the assert() macro will generate no code 
at all. In other words, when you build for testing, the assert is there, but 
when you build the fully optimized version the assert will be removed anyway.

Seems safer to leave the FLAC_ASSERT() and then work to make sure the tests 
cover the potential memory leaks as well as ensure that fully-optimized release 
builds turn off all assert() macros.

Brian Willoughby
Sound Consulting

p.s. I haven't looked too closely at this specific commit change, but speak 
from general experience.


On Jul 4, 2015, at 4:30 AM, lvqcl lvqcl.m...@gmail.com wrote:
 About the removed assert in this commit: 
 http://git.xiph.org/?p=flac.git;a=commitdiff;h=bc5113007a53be2c621d5eb5f4485eddf947ef37
 
 It looks reasonable that if x.num_comments == 0 then x.comments is also NULL.
 Otherwise there's probably a leak somewhere that should be fixed.
 
 
 
 I found several places where the situation is reverse:
 comments can be 0 but num_comments is not; IMHO it makes sense
 to fix them even if there are no crashes (yet?).
 
 The patch is attached (of course it can't fix the removed assert because it
 doesn't fix any leak).num_comments.patch

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] keep qlp coeff precision such that only 32-bit math is required

[Brian Willoughby]

Ok, I just did a comparison of 1.2.1 with 1.3.2, and the change you're 
suggesting was already there before. So, now the question becomes: why was the 
code changed in the first place? Was there a bug that was fixed by changing 17 
to 16, or did someone just get overzealous in a code review and thought that 17 
was a bad choice? Perhaps 32 bits isn't actually large enough to handle the 
overhead of processing 17 bit values - I haven't done the analysis to say 
whether it's safe or not. I do recall discussions that might have been 
referring to this sort of thing, so look at the change log to see why the 
change came about.

Brian Willoughby
Sound Consulting


On Apr 18, 2015, at 12:20 PM, Brian Willoughby bri...@audiobanshee.com wrote:
 Hmm, I don't know the history of the code, but flac 1.2.1 stream_encoder.c has
 
   
 min_qlp_coeff_precision = FLAC__MIN_QLP_COEFF_PRECISION;
   /* try 
 to ensure a 32-bit datapath throughout for 16bps(+1bps for side channel) or 
 less */
   
 if(subframe_bps = 17) {
   
 max_qlp_coeff_precision = min(32 - subframe_bps - lpc_order, 
 FLAC__MAX_QLP_COEFF_PRECISION);
   
 max_qlp_coeff_precision = max(max_qlp_coeff_precision, 
 min_qlp_coeff_precision);
   }
 
 in process_subframe_(), whereas evaluate_lpc_subframe_() has the comments 
 that you refer to.
 
 I've not done an analysis to determine whether these constants should match, 
 or if maybe they should remain distinct.
 
 Brian Willoughby
 
 
 On Apr 18, 2015, at 8:05 AM, lvqcl lvqcl.m...@gmail.com wrote:
 stream_encoder.c has the following code:
 
 
  /* try to keep qlp coeff precision such that only 32-bit math is 
 required for decode of =16bps streams */
  if(subframe_bps = 16) {
 ...
 
 
 But FLAC can convert 16-bit input to 17-bit if mid-side coding is used.
 So, does it make sense to compare subframe_bps with 17?


___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] flac (encoder, analyzer, decoder) tool now (also) available in JavaScript‏

[Brian Willoughby]

If there is a problem with running out of memory, decoding a file in whole will 
surely run into problems. I have FLAC files that are hours long, so mobile 
devices would surely run out of memory. In contrast, a streaming player would 
be able to play any size file.

You were also concerned about licensing restrictions, so it might be better to 
write your own streaming player. Of course, you'd still have the license 
restrictions of the libFLAC or libFLAC++, so that might not really help (I'm no 
expert on open source licenses).

Then again, I understand that there's never enough time to do everything. I'm 
still hoping to find time to write a streaming plugin for iTunes that supports 
FLAC.

Brian


On Jan 20, 2015, at 12:12 PM, Rainer Rillke ril...@wikipedia.de wrote:
 Yeah, de-/encoding a stream would have a lot of advantages but there is no 
 streaming en-/ decoder I would be aware of and for the application I'd intend 
 to use it for, it might be sufficient to de-/ encode a file in whole. 
 Dependent of the time and efforts for creation and maintenance of a stream 
 encoder, it might not fit into the time budget. (Apart from that, as of now, 
 it gives a nice demo application that just runs quickly in every browser, 
 independently from any platform and has nearly zero maintenance cost.)
 
  From: Brian
  
  I'm not sure that I understand your goal. In a browser setting, I would 
  think that you want to decode a stream, not a file, because streaming 
  should have less of a memory impact. The reference decoder is a file 
  decoder, so you probably don't want to port that. I don't think that there 
  is a reference streaming decoder, so you'd need to write your own using the 
  libFLAC or libFLAC++ routines.
  
  Brian
 
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] [PATCH] for flac/decode.c

[Brian Willoughby]

On Dec 14, 2014, at 10:02 AM, lvqcl lvqcl.m...@gmail.com wrote:
 Currently the header of a decoded WAV file can be different to the
 original WAV file because FLAC doesn't preserve 'fmt ' chunk.
 
 For example: create a 24-bit stereo .wav file with WAVEFORMATEXTENSIBLE
 header with channel mask == 3. Encode it to .flac then decode back to .wav.
 FLAC creates a 24-bit stereo .wav file with WAVEFORMATEX header.

Did you try this when using the FLAC command-line option 
--keep-foreign-metadata?

I did not write the code for that option, but I did write a proposal for a 
method that would have allowed the header to be preserved for either WAV or 
AIFF. I'm not sure what was implemented, though, and I have not tested whether 
abnormal WAV headers are preserved.

If --keep-foreign-metadata is not used, then nothing is preserved from the 
original WAV except the audio.

Brian Willoughby

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] WAVEFORMATEX/TENSIBLE

[Brian Willoughby]

On Dec 14, 2014, at 5:23 PM, lvqcl lvqcl.m...@gmail.com wrote:
 Brian Willoughby wrote:
 Currently the header of a decoded WAV file can be different to the
 original WAV file because FLAC doesn't preserve 'fmt ' chunk.
 
 For example: create a 24-bit stereo .wav file with WAVEFORMATEXTENSIBLE
 header with channel mask == 3. Encode it to .flac then decode back to .wav.
 FLAC creates a 24-bit stereo .wav file with WAVEFORMATEX header.
 
 Did you try this when using the FLAC command-line option 
 --keep-foreign-metadata?
 
 Yes. I created a 24-bit stereo .wav file with SoX. It writes
 WAVEFORMATEXTENSIBLE 'fmt ' chunk and a 'fact' chunk. FLAC keeps
 'fact' chunk but writes WAVEFORMATEX header.

Thanks for checking this out.

My vote: If --keep-foreign-metadata does not preserve the header of a 
RIFF/WAVE file (the header is actually a special chunk, 'fmt ', that does not 
always appear at the head of the file), then FLAC should be fixed so that 
this is also preserved when --keep-foreign-metadata is used. It should be as 
simple as saving the 'fmt ' chunk along with the others. If it's already being 
saved, then it should be even simpler to restore the header when decoding.

Brian Willoughby
Sound Consulting

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] [PATCH] for flac/decode.c

[Brian Willoughby]

On Dec 14, 2014, at 1:14 AM, Erik de Castro Lopo mle...@mega-nerd.com wrote:
 lvqcl wrote:
 Also, I have a question.
 
 Currently flac complains about 24-bit .wav files if they have
 old WAVEFORMATEX header and not 'proper' WAVEFORMATEXTENSIBLE
 header. However it writes such files itself.
 
 Is it better to fix this so it decodes 24-bit .flac files to .wav
 files with WAVEFORMATEXTENSIBLE header? Or is it better to leave
 things as is (for better compatibility with old programs)?
 
 That's a good question. I have no idea how to answer that.
I've never quite understood why FLAC acts this way. I use AIFF more than WAVE, 
so I largely ignored it.

My vague recollection is that there were historical problems with the 
specification for WAVE, such that different programs had different ideas about 
what was a valid WAVE file. Things like 24-bit, which were not used much, if at 
all, in the early days of WAVE, were not officially part of the spec. I always 
assumed, or even remembered discussions, that FLAC does what it does for 
maximum compatibility with existing programs.

We probably need some serious research into other programs and their treatment 
of WAVE. What can be read, and what is written by other software (and 
firmware). Ideally, FLAC would accept as wide as possible of a range of 'fmt ' 
chunks, while writing the most compatible variation.


 Maye what's needed is (another) command line flag.

Depending upon whether we find one official standard, or multiple, a command 
line flag might be useful to select the less common standards for those who 
need them.

Of course, it's certainly useful to have a command-line option to suppress the 
warning, but I thought there already was one.

I mentioned earlier that --keep-foreign-metadata should preserve the 'fmt ' 
chunk along with the rest. But I think preserving the original is a different 
topic than what to write when creating a new WAVE file from FLAC audio.

Keep in mind that it's possible to decode to WAVE after encoding from AIFF, or 
the reverse. So in many cases there is no original to preserve. I've even used 
FLAC as a dirty way to convert between AIFF and WAVE when I didn't have any 
other tools (since I almost always have or need a FLAC version of an audio file 
anyway).

Brian

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] [PATCH] Improve LPC order guess

[Brian Willoughby]

On Dec 3, 2014, at 8:30 AM, Martijn van Beurden mva...@gmail.com wrote:
 Op 03-12-14 om 16:48 schreef Olivier Tristan:
 This patch changes the settings associated with compression
 levels 6, 7 and 8. With this patch, -e is no longer used, but
 instead apodization functions are added. This should improve
 compression with at least 95% of all material. Independent tests
 show that this is probably the case.
 
 As it turns out, the material for which the compression didn't 
 improve is mostly 'quiet' material.

I would like to point out that 'quiet' material is common when making original 
recordings.

I do a great deal of live recording, and the levels are always quiet in order 
to guard against clipping. These live recordings are then archived via FLAC to 
make sure nothing is lost.

There are also field recording units that record directly to FLAC, and they 
will also see relatively 'quiet' recordings compared to fully mastered 
commercial CD music. Even though those recorders might not be updated to the 
latest FLAC sources, I would hope that things are not getting worse for a whole 
category of FLAC users.

Basically, I'm worried that the FLAC sources are being fine-tuned to handle 
only the subset of audio represented by commercial CDs, without regard for 
24-bit or raw tracks that have not been fully mastered. It's interesting to 
note that the original FLAC algorithms performed quite well across the board, 
but recent 'improvements' are sacrificing performance in one area to enhance 
performance in another area. Are these the sorts of things that can be handled 
by expanding the compression options, rather than thwarting existing 
performance?

Brian Willoughby

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] [PATCH] Improve LPC order guess

[Brian Willoughby]

Thank you for the explanation, Martijn!


On Dec 3, 2014, at 9:38 AM, Martijn van Beurden mva...@gmail.com wrote:
 Op 03-12-14 om 18:14 schreef Brian Willoughby:
 I would like to point out that 'quiet' material is common when making 
 original recordings.
 
 This is probably a misunderstanding: I said that the material that suffers is 
 usually quiet, not that all quiet material suffers.
 
 I do a great deal of live recording, and the levels are always quiet in 
 order to guard against clipping. These live recordings are then archived via 
 FLAC to make sure nothing is lost.
 
 So do I, and I included a few unaltered samples with plenty of headroom in my 
 standard test corpus for these improvements.
 
 but recent 'improvements' are sacrificing performance in one area to enhance 
 performance in another area.
 
 That is certainly not the case. The compression level retuning doesn't work 
 beneficial for all material, but it is in this case simply coincidence that 
 it is some quiet material that seems to suffer. The patch attached to this 
 thread you're replying to however, for some reason benefits specifically this 
 kind of sounds.
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Two new CVEs against FLAC

[Brian Willoughby]

On Nov 25, 2014, at 8:27 AM, Declan Kelly flac-...@groov.ie wrote:
 
 On Tue, Nov 25, 2014 at 12:29:33AM -0800, mle...@mega-nerd.com wrote:
 
CVE-2014-9028 : Heap buffer write overflow
CVE-2014-8962 : Heap buffer read overflow
 
 Is it known what other FLAC decoding software or firmware is vulnerable
 to these overflows?
 
 Any software player that was derived from the official FLAC codebase
 probably is, and most active 3rd party developers will probably get a
 new release out soon anyway, even if their code was not vulnerable.

My impression - which may be out of date - is that many software players 
embedded the FLAC command line within their GUI app, and then never updated. 
This was a serious pain when FLAC was actively changing, gaining significant 
new features, and yet the most popular GUI apps were left behind. Some people 
were even left with a bad impression of FLAC because they never tried the 
command line, and thus were stuck with the buggy old versions in their GUI.

You are correct that it is easier for a software player to be updated, but I 
have my doubts that any will be updated.


 Embedded systems with native FLAC playback, such as DVD players and
 portable devices, may never get updated.

You might be surprised. If the device allows for firmware updates in the field, 
then the manufacturer may update their device before your favorite Windows GUI 
FLAC player.

I have the Sound Devices 700 series field recorder that records directly to 
FLAC, and they've updated the firmware at least 6 or 7 times since I bought the 
device. I don't think more than one of the updates had anything to do with 
FLAC, but I don't feel worried that they're unable to correct a vulnerability.

Granted, consumer DVD players might not get a firmware update, but these days 
even car stereos have a way for users to get the latest updates. Players like 
the OPPO surely have firmware update capabilities, if needed.


While we're on the topic, what sort of consequences are there, really, with 
this vulnerability? Worst case, your player stops playing on a file that cannot 
be played anyway. Yes, it's bad that you have to power-cycle the player to get 
it to restart, but it's not like you can be doing anything else at the same 
time you're playing a bad FLAC. Have I missed something?

Brian

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Two new CVEs against FLAC

[Brian Willoughby]

On Nov 26, 2014, at 1:40 AM, Erik de Castro Lopo mle...@mega-nerd.com wrote:
 
 Brian Willoughby wrote:
 
 While we're on the topic, what sort of consequences are there, really,
 with this vulnerability? Worst case, your player stops playing on a
 file that cannot be played anyway. Yes, it's bad that you have to
 power-cycle the player to get it to restart, but it's not like you
 can be doing anything else at the same time you're playing a bad FLAC.
 Have I missed something?
 
 I think you are underestimating what a motivated cracker can do starting
 with a simple heap overflow. See:
 
http://en.wikipedia.org/wiki/Heap_overflow
 
 Erik

My point was specifically about embedded FLAC running on a device like a 
player. I should have pointed out that I meant that there is no Linux or other 
operating system, there is no 'root' account, and there are no other programs 
running. The only data structures that exist besides the playback engine would 
be the FAT file system for external storage of recordings.

Besides pure maliciousness, a hacker has nothing to gain by creating a bad FLAC 
that will cause a player to crash.

Brian

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] New release

[Brian Willoughby]

I agree with Miroslav. It is a good practice to make a security release on a 
branch of the stable, shipped code, so that people can obtain the security 
fix with minimal risk to other changes. Even if the new code passes all tests 
fairly soon, it wouldn't hurt to have a couple of releases - one purely for 
security, the next with new changes in other areas.

Brian Willoughby


On Nov 24, 2014, at 12:47 AM, Miroslav Lichvar mlich...@redhat.com wrote:

On Sun, Nov 23, 2014 at 02:44:00AM -0800, Erik de Castro Lopo wrote:
 lvqcl wrote:
 
 I have a couple of questions:
 
 1) Do you plan to release 1.3.1 pre1, pre2 etc or just 1.3.1 w/o any 
 pre-releases?
 
 I had not planned to do a pre-release.

FWIW, considering how much code has changed since 1.3.0, I'd rather
see the security bug fixed in a new 1.3.0 release, maybe with some
other serious bugs like the metaflac memory corruction, and have a
prerelease for 1.3.1 to test it thoroughly.

I know the new release is almost ready, but if some serious bug is
found in 1.3.1, a new release will have to be made anyway to not force
the users to the vulnerable version.

-- 
Miroslav Lichvar
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Compression failed message

[Brian Willoughby]

On Feb 1, 2014, at 00:06, lvqcl wrote:
 Erik de Castro Lopo wrote:

 Sure. But maybe it makes sense to write WARNING instead of an  
 ERROR?

 Well its an ERROR because the flac executable will exit with a non- 
 zero
 exit code, so this condition can be caught in for example a shell  
 script.

 If its only a warning, why would the executable return non-zero?

 But why should it return non-zero exit code?

 The input files are valid, all calculations are valid, but FLAC  
 returns an error...
 IMHO it's counter-intuitive: I can't find another lossless encoder or
 general-purpose file archiver that works in the same way.


It makes sense to have the option to return non-zero when the  
compression fails to compress. As Erik pointed out, a script  
could use the return code to decide to delete the larger FLAC output  
file and keep the original input file since it is smaller (and  
equally lossless).

However, I agree that it is rather strange to return non-zero by  
default, requiring a command-line option to defeat. I would expect it  
to be the reverse: off by default, and enabling non-zero on larger  
files via command-line option.

Brian Willoughby
Sound Consulting

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Compression failed message

[Brian Willoughby]

On Jan 31, 2014, at 23:06, Erik de Castro Lopo wrote:
 lvqcl wrote:

 Sure. But maybe it makes sense to write WARNING instead of an  
 ERROR?

 Well its an ERROR because the flac executable will exit with a non- 
 zero
 exit code, so this condition can be caught in for example a shell  
 script.

 If its only a warning, why would the executable return non-zero?


I think a more appropriate term would be FAILURE, as in COMPRESSION  
FAILURE, because the compression failed to make the file smaller.  
Error implies that the program is not working the way that it is  
supposed to, especially when it suggests contacting the developers to  
correct the problem.

There is nothing to be done with a lossless algorithm that cannot  
guarantee it will always compress any data that might be given to it.  
It is not an error when that algorithm fails to losslessly compress a  
particular set of data. It's simply a mathematical fact.


 Or to compare compress ratio with 1.01 (or 1.05 or 1.1) instead of  
 1.0?

 As soon as the output file is the same size of bigger than the input,
 the encoding process is no longer providing compression. I think  
 checking
 against 1.0 is in fact the correct behaviour.

 There is a --no-error-on-compression-fail command line option. Its  
 documented
 in the HTML docs I've just added it to the --help output.

Ah, there you see that fail is the terminology used in the option.

I suggest replacing ERROR with FAILURE, and removing the comments  
about contacting the developers.

Brian Willoughby

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Exact FLAC subset constraints

[Brian Willoughby]

Yep, that looks conclusive. Thanks for finding this.

Brian


On Jan 12, 2014, at 11:08, lvqcl wrote:

Also found this:

http://lists.xiph.org/pipermail/flac-dev/2008-May/002550.html
http://lists.xiph.org/pipermail/flac-dev/2008-May/002559.html

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Exact FLAC subset constraints

[Brian Willoughby]

I think you've found a bug, Bart.

flac 1.2.1 did not have any FLAC__format_blocksize_is_subset()  
function, so the source you're seeing in format.c must be new. If the  
format documentation you linked to is correct, then either the  
specifications are self-contradictory, or the code is not  
implementing the subset checks as described. My interpretation is  
that blocksize bits of 0110 and 0111 would not be included in the  
subset. The whole point of the subset is to have a small and finite  
list of values so that embedded decoders do not have to deal with  
every possible block size.

This bears further investigation.

Brian


On Jan 6, 2014, at 05:47, Барт Гопник wrote:

I mean that the first statement [Subset streams must use one of
192/576/1152/2304/4608/256/512/1024/2048/4096 (and 8192/16384 if the
sample rate is 48kHz).] published on
https://www.xiph.org/flac/ 
documentation_tools_flac.html#flac_options_blocksize
page IS NOT EQUAL to second statement [The blocksize bits in the frame
header must be 0001-1110. The blocksize must be =16384; if the sample
rate is = 48000Hz, the blocksize must be =4608.] published on
https://www.xiph.org/flac/format.html#subset page.

What statement (first or second) is right?

0001-1110 mean 0110 and 0111 too??? (0110 mean get 8 bit
(blocksize-1) from end of header, 0111 mean get 16 bit (blocksize-1)
from end of header)

Why you don't use STRICT block size checking in
FLAC__format_blocksize_is_subset() like this:

FLAC_API FLAC__bool FLAC__format_blocksize_is_subset(unsigned
blocksize, unsigned sample_rate)
{
 if(blocksize == 192 || blocksize == 576 || blocksize == 1152
|| blocksize == 2304 || blocksize == 4608 || blocksize == 256 ||
blocksize == 512 || blocksize == 1024 || blocksize == 2048 ||
blocksize == 4096 || (sample_rate  48000  (blocksize == 8192 ||
blocksize == 16384)))
 return true;
 else
 return false;
}

instead of

FLAC_API FLAC__bool FLAC__format_blocksize_is_subset(unsigned
blocksize, unsigned sample_rate)
{
 if(blocksize  16384)
 return false;
 else if(sample_rate = 48000  blocksize  4608)
 return false;
 else
 return true;
}

FLAC__format_blocksize_is_subset from format.c IS NOT EQUAL to my
code. E.g. FLAC__format_blocksize_is_subset(1536, 44100) from format.c
returns true, but 1536 is not subset blocksize because 1536 is not one
of 192/576/1152/2304/4608/256/512/1024/2048/4096!

 I don't understand what it is you don't get about those blocksizes.  
 For
 subset streams, the blocksize has to be one of 576/1152/2304/4608 or
 256/512/1024/2048/4096 if the samplerate is lower then or equal to
 48kHz, if higher, 8192 and 16384 are allowed too. If you use any other
 blocksize, the stream is not subset compliant.

 Considering the fixed or variable blocksize stuff, the subset does not
 restrict that, so using a variable blocksize is technically subset
 compliant. However, flake does warn when using variable blocksizes  
 that
 the stream is not subset compliant. The thing is, it's not in the
 reference encoder, so probably most (hardware) decoders haven't been
 tested with it. If you want to be safe, you should probably restrict
 yourself to a fixed-blocksize stream.

 Finally, ffmpeg level 12 is not subset compliant.


 I'm misleading about FLAC subset constraints... Please help me
 understand exact FLAC subset limitation.

  From https://www.xiph.org/flac/ 
 documentation_tools_flac.html#flac_options_blocksize:

 Subset streams must use one of
 192/576/1152/2304/4608/256/512/1024/2048/4096 (and 8192/16384 if the
 sample rate is 48kHz). The reference encoder uses the same block  
 size
 for the entire stream.

  From https://www.xiph.org/flac/format.html#subset:

 The blocksize must be =16384; if the sample rate is = 48000Hz, the
 blocksize must be =4608.

 Finally, one of 192/576/1152/2304/4608/256/512/1024/2048/4096 or
 just (simply) =4608, one of 8192/16384 or just =16384?

 Maximum (4608) value and any intermediate values are not one of
 192/576/1152/2304/4608/256/512/1024/2048/4096.

 The first statement is more restricted.

 Is the word must to be interpreted as described in RFC 2119
 (http://www.ietf.org/rfc/rfc2119.txt)?

 The second question:

  From https://www.xiph.org/flac/documentation_format_overview.html:

 The reference encoder uses a single block size for the whole stream
 but the FLAC format does not require it.

 Should stream MUST have constant (fixed) block size (if blocksize
 satisfies all restrictions from first question) to be subset?

 Is variable block size stream (if blocksize satisfies all  
 restrictions
 from first question) be subset?

 The third question:

 Is FLAC files compressed by FFMPEG with -compression_level 12  
 switch
 are subset?
___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] PATCH for rice_parameter calculation

[Brian Willoughby]

Or, I was originally thinking:

rice_parameter = 0; k = partition_samples;
if (k  mean) {
int n = mean - k;
rice_parameter += n;
k = n;
}

(sorry for the hasty post)


On Oct 11, 2013, at 10:34, Brian Willoughby wrote:
 Hmm, maybe I'm missing something, but what about this:

   rice_parameter = 0; k = partition_samples;
   int n = mean - k;
   if (n  0) {
   rice_parameter += n;
   k = n;
   }

 I've not looked at this code in its context within stream_encoder.c,
 so it's easily possible that I left out something.

 Brian Willoughby
 Sound Consulting


 On Oct 9, 2013, at 08:54, lvqcl wrote:
 MSVS profiler shows that the following code in stream_encoder.c takes
 several percent of CPU time:

 for(rice_parameter = 0, k = partition_samples; k  mean;
 rice_parameter++, k = 1)
 ;

 this code is equivalent to:

 rice_parameter = 0; k = partition_samples;
 while(k  mean) {
 rice_parameter++; k = 1;
 }

 The idea was to accelerate it:

 rice_parameter = 0; k = partition_samples;
 while(k*2  mean) {
 rice_parameter+=2; k = 2;
 }
 while(k  mean) {
 rice_parameter++; k = 1;
 }

 or:
 rice_parameter = 0; k = partition_samples;
 while(k*4  mean) {
 rice_parameter+=3; k = 3;
 }
 while(k  mean) {
 rice_parameter++; k = 1;
 }

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] What is in directory flac.pbproj?

[Brian Willoughby]

On Apr 30, 2013, at 13:45, Marcus Johnson wrote:
 I'd love to fix the .Xcodeproj (which is what they're called now)  
 but I don't have a clue how to, if I were to figure it out would  
 you guys accept it, or is it not worth my time?

Thanks for the offer, but I don't think it's worth the time. Here is  
my reasoning:

1) Historically, flac has been built from the command line on OSX,  
just like every other Unix operating system. In that respect,  
the .pbproj and .xcodeproj files are not really necessary. In fact,  
that's probably why they became outdated and broken, since nobody  
really needed them. I have been working with flac for a long time,  
and I've never seen those files work. When I built the last flac  
release for OSX distribution, I did not use them, preferring to use  
make.

2) If anyone is going to fix the .pbproj or .xcodeproj files, I  
highly recommend that it be done with Xcode itself. Attempting to  
manually edit these files is pointless. You need to verify them with  
Xcode anyway, so you might as well use Xcode to fix them. Frankly,  
recreating them makes more sense than fixing the existing files,  
although you might look in the existing files to make sure you don't  
leave anything out.

3) In my opinion, it is better to create the project files with the  
oldest version of Xcode that is up to the task and supported. The  
reason is that newer versions of Xcode can open old project files,  
converting them if necessary. Unfortunately, the reverse is not true,  
so using Xcode 4 to fix these files would needlessly exclude older  
versions. The only reason to orphan developers this way is when the  
sources make use of new OSX features that are not available on older  
releases, but flac doesn't depend on any OS features that are  
exclusive to OSX, and certainly not on any OSX features that are only  
in newer releases.

4) That said, Project Builder (.pbproj) is simply too old to be of  
use to anyone, so far as I know. Apple possibly still supports OSX  
10.4 Tiger, but anything older than that is in the land of fend for  
yourself (where command line make is going to work fine anyway). I  
believe it is true that Project Builder was last seen on OSX 10.3  
Panther, so I don't have a problem with dropping it. I do maintain  
several Macs running various different OSX releases, and I regularly  
build things under Tiger to ensure that the largest number of people  
can run the binary, but even as old as these machines are it's still  
Xcode, not Project Builder.

I'd like to take a stab at creating a set of .xcodeproj files that  
work with Xcode 2.5 and newer, but maybe someone will beat me to it.

I fully support Erik's request to simply delete the .pbproj directory  
and its contents.

Brian Willoughby
Sound Consulting

p.s. Unfortunately, I do not know who originally created  
those .pbproj files, why they did it, or whether they ever worked.  
You might even find archived list messages where I asked about them.

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Patch to add Unicode filename support for win32 flac

[Brian Willoughby]

On Mar 17, 2013, at 03:57, LRN wrote:
 /me looks at chmod and utime wrappers:
 Ah, i knew i've missed something! :)
 Also, i didn't consider wildcards (i thought shell was supposed to
 handle them...).


I believe that shell does handle wildcards on all Unix variants,  
including OSX. Since Windows does not handle them, I suggest that the  
main flac code not be littered with code that's not necessary on the  
primary platforms. Aren't Windows users accustomed to this feature  
being missing anyway?

Brian

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Higher compression modes from Flake

[Brian Willoughby]

On Mar 14, 2013, at 13:24, Declan Kelly wrote:
 I want the tightest possible compression, while remaining 100%
 compatible with the subset that all known FLAC decoders can  
 successfully
 stream or play now in cars, Hi-Fi units, MP3 players and cell  
 phones.
 The out and out most widely supported lossless audio format could (and
 should) have a better bang for the buck to the average user (who has
 possibly been tempted away from MP3 or WMV or some Apple format).

I have a vague recollection that going beyond -4 is incompatible with  
certain hardware players. Sorry I don't have a reference for this,  
but it seems like even -8 or --best are not 100% compatible with all  
decoders. Going beyond that to -9 or even -12 seems like it would be  
far less than 100% compatible.

Of course, the logical approach is to look into why these decoders  
can only handle some FLAC encodings, and work within those restrictions.


 I buy a lot of music on Bandcamp (and similar sites) and usually get
 smaller files (for long term storage) when recompressing (flac -8).
 A common sentiment I have seen online is my CPU time is too  
 valuable to
 bother with maximum compression, but that ignores the fact that  
 all of
 the copies made of those files are going to add up to something  
 bigger.

I buy nearly everything in FLAC. I do not re-compress, and as far as  
I know there is no easy way to determine how the originals were  
compressed. Download speeds are faster, and I figure that my backups  
are already small enough using the original FLAC that I don't need to  
experiment with re-compressing in the hopes of getting the files even  
smaller.

For original recordings, I use --best, but I'm not concerned with  
direct playback on all devices. If I have a problem with playback,  
I'd probably decode and re-encode. Also, my original recordings are  
raw and unmastered, so I rarely listen to them without mixing and  
processing. I then compress the mastered audio (for CD or HD audio)  
using FLAC again, as a backup of the final product. Again, I use -- 
best and --no-padding because it's convenient, but I wouldn't mind  
learning a better default command line.

Brian

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Higher compression modes from Flake

[Brian Willoughby]

Are you sure that the encoding library was improved, or just the  
command line?

Keep in mind that 1-8 (or 0-8) are just macros for particular  
combinations of options that are also available separately.

I'm just guessing, here, but 9-12 might be nothing more than selected  
combinations of options that are already available in the official  
flac command-line, albeit without a short, numerical abbreviation.

Brian Willoughby
Sound Consulting


On Mar 13, 2013, at 02:49, Marko Uibo wrote:
 Is it planned someday to implement additional higher (9-12)  
 compression
 modes like in Flake?

 http://flake-enc.sourceforge.net/

 It's not very important I think. Harddrive space isn't problem today.
 But it preserves other indepentent work done on Flac and give room for
 some extra albums on harddrive. I think development of flake is  
 stopped.

 I plan to test my ~20 GB flac collection how big difference flake -12
 gives me. But I still use reference implementation for my main  
 collection.


___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] I reinstalled OS X, now FLAC 1.3 git won't compile.

[Brian Willoughby]

When you reinstalled OSX, did you also reinstall the developer tools?  
I often forget that extra step. I assume you did, otherwise you  
wouldn't have gcc or make, but I figure it's worth asking. Also,  
there are option Unix tools that usually aren't installed unless you  
ask for them.

Brian Willoughby
Sound Consulting


On Mar 11, 2013, at 20:36, Marcus Johnson wrote:
 Are there any dependencies that I need, but don't have? I've got  
 doxygen, libogg, automake, autoconf, libtool, valgrind, docbook,  
 nasm, yasm, libiconv.

 the Autogen.sh script fails with:

 Updating build configuration files for FLAC, please wait
 configure.ac:308: warning: macro 'AM_ICONV' not found in library
 configure.ac:309: warning: macro 'AM_LANGINFO_CODESET' not found in  
 library
 glibtoolize: putting auxiliary files in `.'.
 glibtoolize: linking file `./ltmain.sh'
 glibtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'.
 glibtoolize: linking file `m4/libtool.m4'
 glibtoolize: linking file `m4/ltoptions.m4'
 glibtoolize: linking file `m4/ltsugar.m4'
 glibtoolize: linking file `m4/ltversion.m4'
 glibtoolize: linking file `m4/lt~obsolete.m4'
 configure.ac:308: warning: macro 'AM_ICONV' not found in library
 configure.ac:309: warning: macro 'AM_LANGINFO_CODESET' not found in  
 library
 configure.ac:308: error: possibly undefined macro: AM_ICONV
   If this token and others are legitimate, please use  
 m4_pattern_allow.
   See the Autoconf documentation.
 configure.ac:309: error: possibly undefined macro: AM_LANGINFO_CODESET
 autoreconf: /usr/local/Cellar/autoconf/2.69/bin/autoconf failed  
 with exit status: 1
 Marcuss-MacBook-Pro:flac Marcus$ ./configure
 configure: error: cannot find install-sh, install.sh, or shtool in  
 . ./.. ./../..


___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] [flac 1.3.0pre2 pre-release] Updates to test suite

[Brian Willoughby]

On Mar 11, 2013, at 21:37, Erik de Castro Lopo wrote:
 Jaren Stangret wrote:

 Hi all,

 I'm currently going through the massive test suite, updating all the
 scripts making sure they conform to the POSIX standard.

 I've added a bunch of comments to them and have slightly changed/ 
 reworked
 some of the functions to make it easier to read and process.

 I'm also eliminating the use of 'echo' in favor of 'printf' since  
 echo has
 many unspecified uses.

 Attached are only three patches (each patch is for a different test
 script).  If everyone is happy with these three patches, I'll  
 continue and
 rework the rest of the scripts.

 I'd normally say no to a big set of patches like this so close to a  
 final
 release, but if they only touch tests scripts and when I test them  
 they work
 on Linux, FreeBSD and OpenBSD, then I'm happy to have them.

 I'll test this first batch within the next 8-12 hours.

 Erik


Hmm, if this huge patch breaks the tests, how do we know whether  
1.3.0 is performing identically to 1.2.1?

I'm thinking that there is a possibility that the tests might produce  
a false positive. It seems safer to only change the library or the  
tests, but not both at the same time. Maybe the patches to the tests  
should wait until 1.3.1

Brian Willoughby
Sound Consulting

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] 2GB limit patch

[Brian Willoughby]

On Mar 4, 2013, at 20:03, nu774 wrote:
 (2013/03/05 12:27), Cristian Rodríguez wrote:
 Nothing against you code, that's ugly ..but as it is a first step  
 it can
 be forgiven :-)

 I will just force flac to be built with 64 bit file offsets and just
 reject any caller trying to include/link libflac into a non-lfs  
 program
 like

 http://ac-archive.sourceforge.net/largefile/off_t_headers.html

 Then we can fix all the internal problems, without the need of
 typedef'ing around ;)

 Well, I'm afraid typedef'ing or something is mandatory for large file
 support on win32, since off_t is *always* 32bit on win32 and flac
 basically uses off_t everywhere for seek offset and file size.


I do not see the need to jump to 64-bit merely to get beyond the 2 GB  
limit. There are modern API that handle 32-bit unsigned file lengths  
- i.e., up to 4 GB - without going to 64-bit.

Besides, RF64 is the proper format for files that exceed 4 GB, while  
standard RIFF/WAVE should allow reading of files up to 4 GB.

For maximum compatibility, writing WAV files over 2 GB should be  
avoided whenever possible, since so many tools used signed 32-bit  
file sizes.

Brian Willoughby
Sound Consulting

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] 2GB limit patch

[Brian Willoughby]

On Mar 4, 2013, at 22:50, Erik de Castro Lopo wrote:
 nu774 wrote:
 (2013/03/05 12:27), Cristian Rodríguez wrote:
 Nothing against you code, that's ugly ..but as it is a first step  
 it can
 be forgiven :-)

 I will just force flac to be built with 64 bit file offsets and just
 reject any caller trying to include/link libflac into a non-lfs  
 program
 like

 http://ac-archive.sourceforge.net/largefile/off_t_headers.html

 Then we can fix all the internal problems, without the need of
 typedef'ing around ;)

 Well, I'm afraid typedef'ing or something is mandatory for large file
 support on win32, since off_t is *always* 32bit on win32 and flac
 basically uses off_t everywhere for seek offset and file size.

 This requires an API change and hence is something that will need to
 wait for the next release.


This thread started on the topic of WAV files exceeding 2 GB. The  
FLAC API would only need to change if the FLAC file size limits were  
altered. Besides, I believe that FLAC is supposed to be a stream with  
no maximum size limit. No API changes should be necessary for the  
command-line FLAC to support 4 GB WAV files. Did I miss something?

Brian Willoughby
Sound Consulting

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Commonly getting FLAC__STREAM_ENCODER_VERIFY_MISMATCH_IN_AUDIO_DATA on valid audio

[Brian Willoughby]

2013/2/8 Erik de Castro Lopo mle...@mega-nerd.com:
 Collin wrote:

 It turns out it was an error on my part; sorry for the trouble. I  
 misunderstood
 the format libflac was expecting my samples in. I found the  
 wording in the
 documentation very confusing.

 My audio file has 2 channels and 16-bit samples, so I have to pull  
 out each
 individual sample, push it into a FLAC__int32, and fix endian-ness  
 before I can
 pass it to libflac.

 The documentation made me think that libflac just wanted me to  
 shove my 2-byte
 sample into a FLAC__int32. I wasn't aware that it wanted me to  
 extend the sign,
 as well.

 At some stage it might make sense for us to make the FLAC API a little
 more user/programmer friendly.

I seem to recall that the FLAC library requires fixed point numbers  
that are right-justified in a 32-bit fixed-point 'long int', whereas  
the standard everywhere else seems to be left-justified. Left- 
justified makes more sense because it allows 16-bit, 20-bit, 24-bit,  
32-bit, and other depths to be more easily converted.

It's been a while since I needed to know this level of detail,  
because I wrote my own ObjC library on top of the C FLAC library  
since I didn't want to use the C++ FLAC library, and once I solved  
the justification of the bits I never had to think about it again.

However, for new programs being written to use the FLAC libraries, I  
totally agree that there should be a friendlier interface (assuming I  
haven't remembered all of this wrong).

Note that the old interface should be maintained for compatibility  
with existing software. The new, friendlier interface can simply use  
a slightly different (member) function name to access the same  
objects. There's no reason both can't be supported, or even float as  
well.


 Something to match libsndfile's ease of
 use.

I do not use libsndfile, so I wouldn't make this comparison.


On Feb 9, 2013, at 03:13, Johnny Rosenberg wrote:
 That would be great!
 I was actually starting my small ”project” with libsndfile, but it was
 done in a way that I didn't agree with, and since I only work with
 FLAC files (mostly 24 bits) I decided to try to go with libflac
 instead, which seems to let me do things the way I want more than
 libsndfile.

 For example, when loading an audiofile with libsndfile, all the
 samples are converted to floating point numbers, which is what I want
 anyway, but the conversion is done by dividing the integers from the
 file with pow(2,bps-1), but when converting back, they multiply with
 pow(2,bps-1)-1, so if you just read and then write, you end up with a
 slightly lower volume (not that you can hear any difference if you do
 it only a few times, but still, it isn't right, AND it is very easy to
 do it right, so why don't they?)…


 Johnny Rosenberg, the beginner


For a beginner, you sure have the math right. Lossless conversion  
should always be handled via a factor with only a single significant  
digit, otherwise you're actually performing a sum. Multiplying by a  
factor with a single (binary) significant digit is equivalent to  
multiplying by 1. pow(2, N) is a valid conversion factor. pow(2, N)-1  
is not. Basically, converting a 16-bit sample using pow(2, 15)-1 is  
the same as summing 15 lossless conversions that happen to be one off  
from the proper lossless conversion.

The 'why' is that libsndfile hopes to solve a problem that doesn't  
always exist. Namely, that scientific synthesis of data in floating  
point often ranges from +1 to -1, but the fixed point output format  
of most sound files does not allow +1 (even though it does allow -1).  
This is due to the nature of twos-complement fixed-point numbers. I  
have no idea why libsndfile handles integer input to integer output  
in this lossy way, but that was a choice made by libsndfile that  
seems beyond the possibility of correcting. It would have seemed much  
simpler to require floating point data synthesis to apply the pow(2,  
N)-1 scaling only as needed, rather than apply it to all audio files  
(most of which come from fixed-point A/D convertors that don't need  
any scaling). In other words, libsndfile is really only correctly  
applicable to a very esoteric set of data which does not include most  
audio.

Regardless of libsndfile's architecture and design, libFLAC should by  
no means duplicate the incorrect conversion. There especially should  
not be an asymmetrical set of conversion factors.

All of Apple's CoreAudio tools use float without lossy conversion.  
The only requirement there is that synthesized data should avoid +1.0  
sample values unless clipping is acceptable or scaling is added  
before conversion.

Brian Willoughby
Sound Consulting

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Commonly getting FLAC__STREAM_ENCODER_VERIFY_MISMATCH_IN_AUDIO_DATA on valid audio

[Brian Willoughby]

Whatever enhancements occur for libFLAC, we certainly should not  
create a situation where some flag gives different results for one  
set of users versus another. Normalization of data should be separate  
from conversion of data. libFLAC should only handle conversion of  
data. If people want a method for normalizing their audio data, then  
they should use a DAW or at least a library like libsndfile.

To avoid bloat, libFLAC should merely handle conversion between FLAC  
compressed and uncompressed samples. Any potential problems where  
synthetic float waveform samples might clip during conversion to FLAC  
should be handled by a separate tool. In other words, the file should  
be processed before it is compressed.

There are industry standards for conversion whereby only exact powers  
of two are used. Anything else doesn't belong in libFLAC. Namely, pow 
(2,bps-1)-1 should not be used or allowed anywhere in libFLAC.

Brian Willoughby
Sound Consulting


On Feb 9, 2013, at 12:50, Erik de Castro Lopo wrote:
 Secondly, the scaling can be switched off don't you? See:

 http://www.mega-nerd.com/libsndfile/ 
 command.html#SFC_SET_NORM_DOUBLE



___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Tag flac as flac 1.2.1_git

[Brian Willoughby]

On Jan 21, 2013, at 22:57, Erik de Castro Lopo wrote:
 IMO, any code change at all, even just whitespace is worthy of its
 own version number. If the md5sum of the source code tarball is
 different it warrants an updated version number.

Why would you release a new version of FLAC if the binary did not  
change (on a given platform)? I realize that whitespace is an extreme  
example, but in that case you don't really need a new version.

The source code is only consumed by the developers. The version  
control system should handle versions of the source code. For  
example, Subversion would assign a new repository version number  
every time someone checks in a source code change. That is enough.  
There's no need to revise the release version with every developer's  
key press.

It seems to me that you have to change more than source whitespace  
before a public release is warranted. A public release of substantial  
changes (not whitespace) would require at least a 0.0.1 increment in  
version number.

 If we can agree on separating the library and utility version
 numbers, then I think we'll have a much better chance of agreeing on
 version numbers.

 Its use in the test suite means it can't be extracted from the
 FLAC library sources without replicating its capabilities which
 would just be silly.
It would be silly to replicate the command line utility, but you  
don't have to replicate it to change its version number, or to allow  
the version to digress from the format library version.

 Not to mention the fact that embedded devices
 without a command line or any other kind of utility won't needlessly
 see version number changes when the format remains the same.

 Not even bug fixes?
What bugs? I'm not aware of any bugs. A centralized list of bugs  
would be great. I'm sure there is one and I just haven't looked at it  
in a while. Is the old bug database still active? Is there a bug  
search template or report showing the serious bugs that need fixing?

 On that
 note, I suppose this means we might want to mark FLAC files with the
 version of the utility that created them, since the format version
 number won't indicate that going forward - perhaps an application
 block would be appropriate.

 Sorry, but I actually think you have this completely around the wrong
 way. FLAC files can be created without the flac command line utility.
 That suggests that FLAC files should be marked with the *library*
 version that was used to create them.
The last suggestion on my part was a nod to the folks who want to  
define the channel mappings in a way that can be detected by players  
when looking at a specific media file. The FLAC format (library) does  
not need to be modified for this, because FLAC merely holds  
independent channels without any care about their order or position  
or labeling.

But the command line utility might transcode channel mappings from  
other formats like WAVE or AIFF. In that case, having the command- 
line utility version number might help define the channel mapping.  
Then again, it would be better to define metadata specifically for  
the purpose of channel mappings rather than encoding the command-line  
version number into the stream. I'll just take this particular  
suggestion off the table, because there are better ways to provide  
the features desired.

To clarify, I was not suggesting that the FLAC format version should  
be replaced by the command-line utility version - that would  
certainly be a bad idea, as you point out. I should have specified  
that I meant adding a secondary version number somewhere in the  
stream (meta blocks) to indicate any assumption about data that isn't  
part of individual channels.

Brian

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Define 6.1 and 7.1 channel mappings

[Brian Willoughby]

On Jan 17, 2013, at 21:41, Ralph Giles wrote:
 On 13-01-17 7:26 PM, Brian Willoughby wrote:
 I vote for documenting the --channel-map option in the --help

 Do you ever use --channel-map yourself, or recommend it to clients?


Professional surround mastering is delivered on very specific media,  
and FLAC is not an option (to my knowledge).

I use FLAC for archival of original recordings, and I document the  
channel order along with the other details of the recording. I  
recommend the same practice to anyone. FLAC works best with mono and  
stereo. Only stereo can take advantage of compression options that  
share channels. More than two channels in a FLAC does not really  
offer any space savings beyond what you would get with multiple mono  
FLAC files. But if you do archive 8-channels recordings, you'd be  
well advised to document more than just the channel mapping.

Actually, there's quite a large world of possibilities. There are  
recording devices, archival methods, media exchange standards for  
mastering, and only after all of those stages is there delivery to  
the consumer. I'm not aware of any surround material being delivered  
in FLAC yet, neither to mastering houses or consumers. It seems that  
surround is mostly limited to Dolby Digital, DTS, and is limited to  
DVD and BD (Blu-ray Disc). I have seen some new things popping up,  
but many of them look really ugly in one aspect or another.

Are you aware of FLAC being used apart from recording devices (1 to 8  
channels), archival (multiple mono and stereo), computer audio, or  
consumer stereo audio? It seems that there are two areas where  
surround FLAC could take off (mastering formats and consumer media).

Brian

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Tag flac as flac 1.2.1_git

[Brian Willoughby]

The flac front-end utility should have its own version number, on a  
separate schedule from the flac library. I can see that we'd be able  
to add features to the utility quite extensively without ever  
changing the file format or the library. I realize that the utility  
has historically shared the library version number, but I see a  
strong case to separate them from each other to free up development  
possibilities.

Apart from fixing some issues with newer compilers, the library is  
the same code. It should probably remain 1.2.1 or advance to 1.2.2 if  
there are actually any significant code changes.

If we can agree on separating the library and utility version  
numbers, then I think we'll have a much better chance of agreeing on  
version numbers. Not to mention the fact that embedded devices  
without a command line or any other kind of utility won't needlessly  
see version number changes when the format remains the same. On that  
note, I suppose this means we might want to mark FLAC files with the  
version of the utility that created them, since the format version  
number won't indicate that going forward - perhaps an application  
block would be appropriate.

Brian Willoughby
Sound Consulting


On Jan 17, 2013, at 09:27, Ralph Giles wrote:
 On 13-01-16 11:10 PM, Erik de Castro Lopo wrote:
 My understanding is that the recent changes for 7 and 8 channels was
 a documentation change only.

 I think we should also change the flac front-end utility to construct
 and interpret the WAVE channel mask for 7 and 8 channel files. No one
 has written that patch yet.

 FWIW I generally agree with Erik that 1.3.x is justified by the long
 period without a point release. Adding a channel mapping for 7 and 8
 channel files is a small spec addition, and doesn't have a serious
 effect on deployed implementations.


___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Tag flac as flac 1.2.1_git

[Brian Willoughby]

The amount of time that has passed since the last change has nothing  
to do with the version number.

I'm inclined to believe that 1.2.2 would be appropriate.

I'm sure there will be other, more appropriate ways to celebrate the  
new release after the long period of stability.

Brian Willoughby
Sound Consulting


On Jan 11, 2013, at 23:56, pyth.flac-dev.5@spamgourmet.com wrote:
 The initial rule was, if I can recall correctly :

 - Changes in the first digit (e.g. 1.x.x to 2.x.x) indicate a break  
 in backwards compatibility ; i.e. the formats are totally different.
 - Changes in the second digit indicate backward-compatible changes  
 in the format (i.e. a 1.1.x-encoded file is only a particular case  
 of a 1.2.x-encoded file)
 - Changes in the third digit reflect any other, non-format related,  
 change. In particular, any 1.2.x decoder can decode any 1.2.y- 
 encoded file.

 I think it best to stick to that, but you're doing the work, so you  
 pick up what you think best or easiest. I believe however it is  
 good to have rules that precisely govern what number changes.

 Cheers,
 Pyt.


 On Sat, Jan 12, 2013 at 8:37 AM, Erik de Castro Lopo - mle+la@mega- 
 nerd.com flac-dev.pyt.682528eb7b.mle+la#mega-nerd@ob.0sg.net  
 wrote:
 pyth.flac-dev.5@spamgourmet.com wrote:

  I seem to recall that changes in the second number indicated a  
 minor change
  in the *format* of the file itself (for example, 1.1.x to 1.2.x  
 introduced
  a new rice coding option used for 24-bit files).

 I wasn't aware of that.

  Are there any format changes that would justify that ?

 I consider the first release in 5 years to be a sufficiently major
 thing to warrant the bump in versions number, but if people thing
 1.2.2 or 1.2.10 or whatever is mor appropriate then I'll go with
 that.



___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Tag flac as flac 1.2.1_git

[Brian Willoughby]

On Jan 12, 2013, at 14:28, Martijn van Beurden wrote:
 On 12-01-13 22:46, Brian Willoughby wrote:
 I would suggest that everyone keep in mind the vast installed base of
 hardware FLAC recorders and players, and not senselessly make them
 obsolete without extremely compelling reasons.

 This can be done for the same reason the change from 1.1 to 1.2  
 added a
 new form of residue coding: I don't believe there are many 7 or 8
 channel FLAC-players out there (just like there were not much 24-bit
 FLAC decoding devices out there when 1.2 came out). We still are at a
 point we can still make those changes.


The problem with your example is two-fold. First, residue coding  
affects the actual compressed audio data, but channel mapping does  
not. Second, there are 8-channel FLAC recorders out there, so it  
would be incredibly destructive to break compatibility for no reason  
at all.


 Josh made a decision at some point to map audio channels instead of
 storing the order in metadata. I think it was a sound decision because
 the channel order is known without the need for that piece of  
 metadata,
 for example in case of streaming. It would be silly to keep this
 behaviour on currently defined channels and do something else on 7  
 and 8
 channel audio.


It seems that you are imagining a decision on Josh's part without any  
reason to believe so. The existing format has channel count and  
channel combination attributes, not channel order variations. The  
information that exists in the headers is not optional, so I don't  
see how you've proven that Josh made any sort of decision here. The  
headers simply must include this information in order to extract the  
uncompressed audio channels.

In the METADATA_BLOCK_STREAMINFO, 7 bits are allocated for the  
channel count. This limits the format to 8 channels in a single  
stream. Nothing is implied about order or channel labels, and there  
is no room to go beyond 8 channels without breaking compatibility.

In the FRAME_HEADER, 8 bits are allocated for the channel  
interpretation. The first 8, codes 0 through 7, refer to independent  
channels, where none of the channels affect any of the other  
channels. Codes 8, 9, and 10 are variations of stereo where at least  
one of the coded channels is a combination of multiple uncompressed  
channels (side), if not both (10: mid/side). Codes 11 through 15 are  
reserved. Although these channel formats include labels, none of them  
is redundant in terms of coding the same format with different  
channel names.

In other words, the FLAC format describes purely how to extract the  
individual channels as audio data streams, without any possibility of  
channel names or labels altering the format. It's merely a convention  
that the multi-channel options mention a default channel order. There  
is no reason to change the format just to change the labels for each  
channel. There is no example yet of two FLAC files with the same  
number of channels and the same data coding for those channels, but  
which only differs in the labeling and output assignment of those  
unaltered channels. If there were, then I'd say you have a case.


 We might try to find out how current 7.1 hardware decoders behave with
 regard to channel mapping.


Yes, that would be a great project.

DVD includes uncompressed PCM, Dolby Digital surround, and DTS  
surround. It would be interesting to summarize the channel orders for  
each of those. PCM is limited to stereo, though. I have DTS encoding  
software, but have never paid attention to the channel order because  
there is no option. They have altered DTS over the years to add  
channels, but always in a backwards compatible manner so that new  
media with additional channels can still be played by old hardware  
that only understands fewer channels.

BluRay discs have even more audio formats, and it would be  
interesting to see a report on channel orders for all of those  
variations.

We have a different situation. If someone were suggesting that FLAC  
be modified to support more than 8 channels, then I would understand  
a format change and requisite version number change. But all we are  
really talking about is keeping the channel counts the same, and the  
raw compressed data, but with additional information on channel  
labels that would facilitate patching the correct audio data to the  
correct physical output wire for analog audio (or maybe digital).


Let me put this another way: Can you make a stronger case for why we  
should NOT create a standard meta data chunk for channel labels?

The existing changes that have been proposed do nothing to change the  
bit stream itself. The only difference is a change from 7 or 8  
unspecified channels of audio to 7 or 8, respective, channels of  
audio with specific labels. That is a change in human interpretation,  
not a change in how the bits are decoded. The same 7 or 8 channels of  
audio data are still created exactly the same

Re: [flac-dev] Bug or strange behaviour or --output-prefix

[Brian Willoughby]

Seems like what you really want is an --input-prefix parameter.
You might also like a --create-output-directories option.

In all cases except absolute paths, the input prefix must be assumed  
to be the current working directory. Therefore, any relative paths in  
input file names must be preserved on output to avoid collapsing  
multiple source directories into a single output directory, with the  
subsequent danger of multiple files with the same name overwriting  
each other.

In the case of absolute paths (and even in some of your other  
examples), you seem to be expecting the program to do some sort of  
automatic analysis of the input prefix, output prefix, and relative  
file paths to look for and eliminate overlapping sub-paths. That's  
something that simply isn't done in Unix command-line utilities, at  
least not in any way like what you describe or imply as desirable.


This is basically the long way of saying that the behavior you see is  
by design - it's intended - and you'll find that all Unix utilities  
work the same way. There is nothing to fix here, although some new  
features might get you what you want.


For example, --input-prefix would allow you to use

flac -d --input-prefix somedir --output-prefix ../ somefile.flac

or

flac -d --input-prefix /home/someuser/somedir --output-prefix /home/ 
someuser somefile.flac

(and don't forget that all variations and combinations of options  
must continue to work predictably when more than one FLAC file is  
given. I often process *.flac in large directories.)


For the second example, --create-output-directories would allow you  
to use any of the examples you gave without having flac stop with an  
error when the output directory structure does not already exist.

Brian Willoughby
Sound Consulting


On Dec 18, 2012, at 07:58, Martijn van Beurden wrote:
 I wanted flac to decode some file to another directory than the
 current. The man page says I should use --output-prefix for that, but
 that doesn't seem to work correctly. For example

 flac -d --output-prefix ../ somefile.flac

 seems to work fine, just as expected. However, the next example

 flac -d --output-prefix ../ somedir/somefile.flac

 doesn't work as expected. the FLAC utility seems to try to decode  
 to the
 directory ../somedir which doesn't exist. Even using absolute paths do
 not work. For example

 flac -d --output-prefix /home/someuser/ /home/someuser/somedir/ 
 somefile.flac

 it seems that it is trying to encode to
 /home/someuser/home/someuser/somedir/somefile.flac. This can't be the
 intended behaviour right? Now I think of it, what I expected was  
 similar
 behaviour to something which I would call --output-directory. The way
 this is coded looks like a quick fix, but to me it seems this is only
 useful when one has to use a file in the current working directory  
 or in
 some special cases where identical directory trees are used.

 So the question is, is this intended behaviour or a bug? If it is
 intended, an --output-directory parameter would be useful for  
 scripting.


___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Git branch with compiling fixes for win32

[Brian Willoughby]

On May 7, 2012, at 06:11, Miroslav Lichvar wrote:
 On Fri, May 04, 2012 at 11:22:00AM -0400, Cristian Rodríguez wrote:
 El 03/05/12 12:19, Miroslav Lichvar escribió:
 It makes the C function faster than the corresponding asm  
 routine, so
 if it's included I'd suggest to just drop the asm function to not  
 keep
 around more asm code than is necessary.

 With current compilers it is very likely that those routines are  
 already
 superflous.

 It seems the current compilers are not that good yet :).

 In a test on a Core 2 machine with gcc-4.6.3, i686 flac build with
 nasm enabled is about 7% faster in decoding than without nasm. x86_64
 build is about 2% faster than the i686 build with nasm enabled.


Thank you for running these benchmarks and reporting on the results,  
Miroslav.

As a sometimes assembly coder, I believe that compilers will never be  
able to beat a human mind in all situations, even if they can in  
most.  I can't speak for the FLAC assembly developers, but when I  
write assembly, I always check the compiler's output at all  
optimization settings just to be sure I'm not wasting my time.  That  
certainly doesn't guarantee that the compiler will never improve its  
results, but keep in mind that changes to compilers sometimes produce  
less optimal results for particular cases, even if they involve  
improvements in most general cases.

But it's certainly good to double-check periodically as compilers  
improve.

Brian Willoughby
Sound Consulting

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] FLAC QUESTION

[Brian Willoughby]

On Apr 12, 2012, at 07:12, Rafael Velasquez wrote:
 I'm using flac for an application.
 The fact is that i'm trying to encode a file that is not necessary  
 a wav file.

 So, at first, i would like to know if it's possible to encode a  
 file different from the wav file with the flac encoder ?


First of all, the FLAC API does not work with WAV format data - you  
must provide raw audio samples.  If you have a WAV file, then you  
should use other code (your own?) to extract the raw audio samples  
from the file, leaving out the chunk headers and other, non-audio  
information.  You should also make sure that the FLAC object is  
initialized to accept the same size samples and channel count as you  
have in your data, and it would probably be a good idea to match the  
sample rate as well for proper decoding later.  Beware that the  
format you have may not exactly match what the FLAC library wants, so  
you may have to alter the data slightly before sending it to the API.

So, it doesn't matter whether you have WAV, AIFF, SDII, or some other  
file, because your code must first extract the raw audio samples from  
the file before calling the FLAC library.

Note that you can encode a WAV file as FLAC, and then later decode  
the FLAC file as AIFF.  FLAC does not usually care what the  
uncompressed foreign file format was (unless you use preserve  
foreign metadata).  These operations are supported by the command- 
line program, but the WAV and AIFF format code is not part of the  
FLAC library.


 When i put my data into : ok =  
 FLAC__stream_encoder_process_interleaved(encoder, pcm, need);

 after some rounds in my boucl i get  :
 encoding: FAILED
state: FLAC__STREAM_ENCODER_VERIFY_MISMATCH_IN_AUDIO_DATA
 (=== what does it really mean ? and what are the reasons to be in  
 this case ? )

 I'm not english spoker, so sorry if i'm not very clear.

There is documentation for the FLAC API.  I do not see anything  
describing error messages, but you could look into the source code.   
My guess is that mismatch refers to the checksum.  I do not have an  
answer for you here, but perhaps you can continue to work on your  
code and find the problem.

Brian Willoughby
Sound Consulting

___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] [PATCH] Fix buffer overflow in metaflac

[Brian Willoughby]

Agreed.  I was going to suggest memcpy() or something equivalent,  
because the FLAC structure is not literally a C string, but rather a  
32-bit field that may or may not have a terminating NULL.  Erik's  
code should work correctly in all cases.


On Apr 5, 2012, at 04:02, Erik de Castro Lopo wrote:
 I actually think that this is a better solution:

 if(strlen(r) == 4) {
 -   strcpy(out-entries 
 [entry].application_id, r);
 +   memcpy(out-entries 
 [entry].application_id, r, 4);
 }



___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [flac-dev] Compiling flac for OSX 10.4

[Brian Willoughby]

Hi Glenn,

Thanks for your efforts to support 10.4

I still run 10.4 on a G5.  I believe that it can sometimes be more  
difficult to compile for 10.4 from newer systems, whereas building  
for 10.4 on 10.4 is a cinch.

Note that the Xcode project may be broken.  I don't think that anyone  
ever really used it.  I recall trying, and it was too broken, so I  
gave up.  What I always do with FLAC is build from the command line  
just like with any other flavor of Unix.

Did you try building FLAC from the command line using the normal Unix  
instructions?

When you build under Xcode, did you build the Release version instead  
of the Debug version?  Sometimes it necessary to make a Deployment  
version or use something equivalent to make install.

Finally, did you try the flac 1.2.1 installer disk image that I  
compiled?  Look on flac.sourceforge.net for flac-1.2.1.dmg - it  
should have everything you need.  If there are any problems with this  
build (and nobody has complained to me since 2009), then please let  
me know so I can make corrections and update the installer.

Brian Willoughby
Sound Consulting


On Feb 9, 2012, at 22:44, Glenn McCord wrote:
 I've been trying to compile flac 1.2.1 on OSX by using the 10.4 SDK. I
 need to use the 10.4 SDK because the application linking to flac
 requires it. Compiling flac with 10.5 results in link issues due to
 10.5's fopen being different.

 As I've spend a ridiculous amount of time on this, including trying to
 compile the source using an xcode project (which did work until the
 app that requires flac gave me all sorts of stack_chk_fail errors
 (http://tinyurl.com/76uegap) ) I thought I'd post here to inquire if
 any one has been able to compile using 10.4 SDK

 Any help with this would be greatly appreciated.


___
flac-dev mailing list
flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] Bug: end-of-line in FLAC console output

[Brian Willoughby]

It's becoming clear to me that we need to distinguish between the  
different aspects of FLAC software, whether it be the core library,  
command-line, or third-party tools.

Although I use the command-line tool exclusively (plus a few of my  
own tools), I really don't care about bugs there or missing  
features.  Maybe I'm being selfish, since my bugs fixes and  
significant design suggestions have all been implemented in the  
official release, so I'm kinda done with the need for change.

Where my concern is focused is the core library.  There I have  
experienced no bugs at all, and I've recorded hundreds of live  
musical performances that were either originally created as FLAC  
(SD702) or compressed to FLAC before archival.

The third-party front ends tend to distribute old versions of the  
command line, and what's most important is that the third-party tools  
were never maintained by Josh.

Glancing at the bug database, it would be helpful if the open bugs  
could be classified, particularly if, as I suspect, many of them  
would end up with an official status of will not fix or by design.

Finally, I also consider porting efforts to be separate.  So, if  
Windows support is the only place where compile errors and bugs are  
appearing, then it would make more sense to me if there was a Windows  
porting project that is distinct from the official FLAC distribution.

Brian


On Nov 21, 2011, at 08:39, Declan Kelly wrote:
 On Wed, Nov 16, 2011 at 03:36:12PM -0800, bri...@sounds.wa.com wrote:
 As a seasoned software developer, I've learned the hard
 way that every single change to a source code repository is a chance
 for a new or old bug to appear.  I am not aware of any bugs in FLAC,
 so the lack of changes is perfect.

 The only bug I'm aware of is the end-of-line one, where the command- 
 line
 utility's output goes one character further than it should,  
 resulting in
 extra scrolling when the output is too long for the console.
 Anyone using a FLAC front-end will not even be aware of that, and I am
 guessing that most possible fixes might break any of the existing  
 front
 end tools that depend on this behaviour.


___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] A-law and mu-law

[Brian Willoughby]

On Nov 21, 2011, at 15:25, Conrad Parker wrote:
 In any case Erik is maintaining both libsndfile and libflac, and it's
 unlikely he'd want to duplicate the code. If anything it'd make more
 sense to remove code for reading other formats from the flac sources
 and just use libsndfile for that part.

It would be a bad idea to remove code from the FLAC command line.  As  
far as I know, there is nothing in libFLAC that deals with WAV or  
AIFF, so there is nothing to remove there.  Only the command-line  
tool deals with external formats, and the command-line tool should  
not be crippled beyond what it is already.

Brian Willoughby
Sound Consulting

___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] Git branch with compiling fixes for win32

[Brian Willoughby]

What bugs?

On Nov 18, 2011, at 04:16, Bastiaan Timmer wrote:
 It's good to see some updates to the FLAC project after so much  
 time! Will there be any timeline for a bugfix-release?


___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] Git branch with compiling fixes for win32

[Brian Willoughby]

On Nov 16, 2011, at 14:11, Declan Kelly wrote:
 On Wed, Nov 16, 2011 at 05:41:21AM -0800, avu...@gmail.com wrote:
 Hate to be Capt. obvious here, but there's a lot of development going
 on here that should be encouraged. If the FLAC project isn't going to
 open up, it would make a lot of sense for someone to take over
 maintenance on a github account with the git-cvsimport or such. I
 don't see anyone stepping up, me included, but I wanted to throw this
 out there in case someone hasn't really thought about it yet.

 Looking at the FLAC website, the most recent news is almost 2 years  
 old,
 and there's no evidence to show that the project's still live. I  
 haven't
 seen any post from Josh Coalson on this list in a while: is he still
 running the project?

 Something else to think about:
 Apple recently released sources for Apple Lossless reference  
 utilities.
 It's all under an Apache license.

 This will (presumably) lead to more software developers spending more
 time improving ALAC support in their projects, with more reference
 material available than the unofficial reverse-engineered code (as  
 used
 in vlc and libavcodec).

 This will leave Apple with even less reasons to support FLAC in their
 own products. Anyone with an iPad/iPhone/iPod must install Rockbox to
 play FLAC files, and the Apple TV can only play FLAC (and anything  
 that
 was not bought from iTunes) using XBMC, after being jailbroken.

 But for the Average End User, they don't want to have to jump  
 through a
 bunch of hoops to get FLAC support. Apple Lossless will Just Work  
 on all
 Apple devices (as it always did), but now it's more freely  
 available so
 the freedom-loving hippies can stop complaining about the source code.


Declan,

You make some important observations, but I do not see how anything  
can be done by the FLAC team about Apple's lack of support.

As for FLAC, no news is good news.  That means the code is stable and  
bug free.  As a seasoned software developer, I've learned the hard  
way that every single change to a source code repository is a chance  
for a new or old bug to appear.  I am not aware of any bugs in FLAC,  
so the lack of changes is perfect.

It seems that all of the recent updates have been efforts to port  
FLAC to operating systems like Windows and Linux.  In terms of your  
comments above, Windows/Linux support does absolutely nothing to help  
FLAC compete against ALAC on the OSX platform.

It is indeed noteworthy that Apple has released the source for ALAC.   
The power of FLAC is that it was designed for embedded systems from  
the beginning, and that's why you see portable recorders like the  
Sound Devices 700 Series supporting FLAC, as well as various optical  
disc players (CD, DVD, etc.).  One question that remains for me is  
whether Apple's ALAC open source can be ported to these kinds of  
embedded systems with the same ease.

Yes, it has always been an issue that Apple never seemed motivated to  
support FLAC as a first class file format in OSX, CoreAudio, iTunes,  
and their hardware platforms based on iOS. I really don't see how  
there is much that Josh Coalson or anyone outside Apple can do about  
this.

If people in the FLAC community have CoreAudio converters for FLAC or  
iTunes plugins for FLAC, then it would be great to see those  
contributed to the open source collection.  However, none of those  
items would really result in a change to the FLAC library sources,  
which have been stable and solid for a respectable amount of time.

Brian Willoughby
Sound Consulting

___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] MD5 Signature Mismatch

[Brian Willoughby]

On Jun 1, 2011, at 05:12, Scott C. Brown 02 wrote:
 He sent me a link to the files here:

 http://www.archive.org/details/wolf2011-05-29.cleantone
There are no uncompressed files here, so it's difficult to discover  
what you need to know.

 I just grabbed the first track (only 2.7 MB), and metaflac gave me  
 an MD5, but
 the file failed testing and won't decode.
At least you've determined that there is an MD5.  I should test one  
of my files with a missing MD5 to see if it's possible to tell the  
difference between a bad MD5 and a missing MD5.

 md5 in flac file: 8cefa4d345df955912a2ca94178f072d
 correct md5 according to shntool: ce67ad906f81db0d914104bea08b0825
I could be wrong, but I don't think that it is possible to calculate  
the correct MD5 without the original file.  FLAC does not calculate  
the MD5 of the compressed data, but instead calculates the MD5 of the  
uncompressed data.  You need the original, uncompressed data before  
you can know the correct MD5.  Otherwise, the uncompressed data that  
comes from the decoded FLAC could be wrong, and would thus give a  
different MD5, possibly even a third value.

 So it looks like his machine is creating bad md5s.  I just don't  
 know enough about
 why this could be the case or what else I should be looking at...

This could be an uploading problem.  There are still too many  
variables at this point to know for sure.  Is he using the flac  
command line tool, or some third-party front-end for FLAC?

Brian

___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] MD5 Signature Mismatch

[Brian Willoughby]

On Jun 1, 2011, at 19:15, Scott C. Brown 02 wrote:
 I thought it might be an uploading problem, but he also gets the  
 same issues when
 simply moving from one of his computers to the other.   (I supposed  
 that could
 also be a transfer issue, though I doubt it).

 He's using a 3rd party front end for the mac (xACT) which simply  
 calls the flac
 1.2.1 command line tool.


I'm starting to suspect xACT.  I've seen problems with the third  
party front ends before.  Personally, I only use the official command  
line for flac, particularly since I deal with original recordings and  
FLAC is my backup format of choice.  I just can't afford to let a  
third-party programmer make mistakes.

FYI: I grabbed an old FLAC from my hardware recording device, and  
metaflac --show-md5sum T99.FLAC says  
 ... so that's what it looks like  
when the MD5 checksum is completely missing.  This is obviously not  
your problem, but I wanted to specifically rule it out since we've  
not had any luck so far.

Brian

___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] MD5 Signature Mismatch

[Brian Willoughby]

I have a hardware device which records directly to FLAC.  I have  
found that the MD5 signature is not created during recording,  
probably because it would delay the process to calculate the MD5 on  
hours of music after you push stop.  It's much easier to calculate  
when converting existing audio rather than when recording new audio.   
Anyway, my point here is that it is possible for a FLAC file to exist  
without an MD5 signature, so my first suggestion is to learn how to  
distinguish between a true mismatch and simply a missing MD5.  I  
must admit that I don't pay attention, since I only get the warnings  
when dealing with FLAC files from live recordings, never from  
converted AIFF or WAVE.

As for why this is happening in the specific situations you've  
described, I have no idea.  My suggestion is to first gather more  
information, by learning how to confirm whether an MD5 Signature even  
exists in these files before continuing to determine the reason for  
the mismatch.

Maybe someone else has more information.

Brian Willoughby
Sound Consulting


On May 31, 2011, at 19:20, Scott C. Brown 02 wrote:

 I found an old thread from 2007:

 http://www.mail-archive.com/flac@xiph.org/msg00702.html

 I have a user who is getting this same MD5 Signature mismatch.

 2 times he told me he got it today:

 1 time: using the flac 1.2.1 encoder (command line compiled from  
 official flac
 src) on a g5 (power pc) mac, he created a file set and uploaded to  
 archive.org.
   archive.org uses flac 1.2.1 to test the files and then convert  
 them to ogg/mp3.
   the test step gave MD5 signature mismatch.

 2nd time: he opened a flac file created on the power pc mac on a  
 windows machine
 in CD wav and split it up into 3 flac files.  he then copied the  
 files back onto
 the power pc mac and tried to decode them.  flac 1.2.1 (command  
 line compiled
 from src) gave an md5 signature mismatch error

 he ran a ram test and Techtool Pro reported his ram to be ok.   
 Anyone have any
 idea what's causing the issue?

 Scott

___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] Can a libFLAC encoder be initialize and called from inside a libFLAC decoder callback?

[Brian Willoughby]

Have you tried decoding to memory, and encoding from that memory?   
This technique would avoid file I/O.

If you are reluctant to allocate enough memory for the full file (and  
why not, with a 64-bit system?), then you could implement a FIFO.   
You would need a semaphore to keep track of the decoder's progress,  
and multithreaded coding.  The decoder would pause, whenever the FIFO  
gets full, until the encoder empties out the samples so far.

It's also possible that you have made a mistake in InitializeEncoder 
() where the new file format does not match the incoming Buffer blocks.

Brian Willoughby
Sound Consulting


On May 24, 2011, at 10:16, David Troendle wrote:
 I am getting large, corrupted native FLAC files when trying to call  
 the encoder from inside the decoder.  The metadata in the output  
 file is OK.

 Is initializing and calling an encoder from inside a libFLAC  
 decoder write callback supported?  The encoder is initialized on  
 the first write callback.

 Here is some context. The static callback calls the instance  
 callback.  Encoder is initialized on line with “m_HaveEncoder =  
 InitializeEncoder();”.  Samples are submitted to the encoder on  
 line with “FLAC__bool OK = FLAC__stream_encoder_process(m_Encoder,  
 Buffer, Blocksize);”.

 I have a version that decodes to a WAV file and then re-encodes the  
 WAV file, but would like to avoid all that I/O.




___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] Can a libFLAC encoder be initialize and called from inside a libFLAC decoder callback?

[Brian Willoughby]

On May 24, 2011, at 12:11, David Troendle wrote:
 Thanks for the tip, Brian.  I did have a version that does  
 everything in
 memory, but only had enough memory to get six threads going.   
 (Although my
 system has 16GB, I have not taken the time to create 64-bit  
 libraries for
 wxWidgets, TagLib, libFLAC, etc.)

What's wrong with only 6 threads?  Is that just 3 FLAC recodes (3  
decode, 3 encode)?  With a queuing layer driving the threads, you  
could schedule hundreds of file recodes and each one would start as  
soon as a thread opens up.


 Based on the direction you are pointing me in, I assume that  
 encoding from
 within the decoder is not permitted.

I have no idea.  My assumption before your email would have been that  
the individual encoder/decoder handles would allow reentrancy, but  
perhaps there are pieces of libFLAC that are not reentrant even when  
working on separate encoder/decoder objects.  The only way to know  
for sure is if Josh Coalson pipes up, or someone takes the time to  
look at the source for a thorough code review.  There's a possibility  
that your code has a bug, and what you want to do may actually be  
possible.  But, again, I have no idea.


 I really like your idea of the FIFO, and will probably go in that  
 direction.
 What do you think of implementing the FIFO via a pipe?  That might  
 simplify
 the implementation.

You don't strictly need a FIFO.  A pipe might be a good choice - at  
the very least it would allow you to link two processes instead of  
being limited to two threads within a process.  The pipe could also  
allow you to decode on one machine and encode on another over a  
network connection - assuming that's of any use to you.

Brian Willoughby
Sound Consulting

___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] Variable Bit Rate

[Brian Willoughby]

On May 23, 2011, at 10:26, Dennis Brunnenmeyer wrote:
Is FLAC a variable bit rate format when streamed? If so, how can it  
be truly lossless?


No offense intended, but your logic is backwards.  You should be  
asking: How can a constant bit rate format be truly lossless ...  
unless it is not compressed at all.


Look at it this way: You can either guarantee a low bit rate, and  
you'll have to lose data to fit ... or you can guarantee that no data  
is lost, and you'll have to adjust the bit rate up to guarantee  
that.  You can't have both the guarantee of a (low) target constant  
bit rate and a guarantee that nothing will be lost when you reduce  
bit rate below uncompressed.


Masklinn actually did a great job of explaining why, I just wanted to  
present a different way of looking at the problem.


Brian Willoughby
Sound Consulting

___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] Variable Bit Rate

[Brian Willoughby]

On May 23, 2011, at 11:35, Dennis Brunnenmeyer wrote:
 I'm well aware how compression works. But images and document files  
 do not depend on the relative timing of the data to reproduce  
 themselves. They are in essence only two-dimensional in space,  
 whereas the data in a sound file is time-dependent.
Images are three-dimensional or maybe five-dimensional,  
mathematically, because the pixel value at each two-dimension point  
can have any value (monochrome) or color (three-dimensional RGB).

Documents and sound files are two dimensional.  You cannot change the  
position or value of a character in a text file without losing  
information.

The key point here is that the timing you refer to in a sound file is  
not really so special.  It is merely another dimension of the data.   
It is preserved in FLAC.  Of the various methods for drawing sound  
files on the screen, they are all at least two-dimensional, if not  
more, which should be a clue that sound files are two-dimensional.


 The question really has more to do with the decoded FLAC stream  
 output, which I presume is a linear PCM file, e.g. WAV.  If FLAC is  
 lossless and created from an original CBR WAV file, is is true that  
 the decoded output is also CBR when played?

 That is, WAV in = WAV out, where both are CBR?
Yes, an uncompressed sound file is CBR, unless you're talking about  
LDPCM.  FLAC is compressed, though, and thus it must be VBR in its  
compressed form.  The Variable in VBR ranges anywhere from slightly  
above the CBR of uncompressed audio (including overhead) to  
approximately half that rate (on average) or even sometimes lower.


 Thanks for any insights on this matter. I've been told that because  
 a FLAC stream from a server to an application is VBR, that certain  
 transients are not handled correctly, like the ringing of bells. If  
 this were true, FLAC would not be lossless in this application.
You have been told wrong.  If such things happen with streamed FLAC,  
then there is a flaw in the streaming software.

One thing to keep in mind is that a VBR format like FLAC requires  
latency when streaming.  If the streaming software is not designed  
with adequate latency, then you could have artifacts when the data  
does not appear in time.  But that is not the fault of the format,  
but rather that the playback is trying to get ahead of the format -  
which is impossible.

Brian Willoughby
Sound Consulting

___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] Variable Bit Rate

[Brian Willoughby]

On May 23, 2011, at 12:09, Dennis Brunnenmeyer wrote:
 Am I right in assuming that in order to deal with potential latency  
 issues,  an application needs a sufficiently large FIFO buffer as  
 well as the proper decoder?
Basically, yes.  It does not need to be strictly labeled as a FIFO,  
but the order should certainly be preserved in whatever buffer is  
used.  The streaming software could be as simple as decoding to a WAV  
file and then starting playback after a sufficient amount of data has  
been collected.

Brian Willoughby
Sound Consulting

___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] Variable Bit Rate

[Brian Willoughby]

On May 23, 2011, at 15:05, Richard Ash wrote:
 I suspect in fact that the difference in volume in that forum post
 explains the change in perceived sound quality completely, human  
 hearing
 being very sensitive to that sort of thing. As to why the volume  
 doesn't
 match, I suspect that one of the playback chains is integer all the  
 way
 and the other is float, with the conversion being not quite right ...

You're probably right, or at least something similar is going on.

I primarily use a FireWire audio interface which allows me to loop  
back the exact digital data that is being sent to the DAC such that I  
can analyze the data.  This comes in very handy for confirmation that  
a particular piece of software is not altering the data along the  
way.  Now that we have 16-bit, 24-bit, and lossy formats (often  
resulting in 32-bit samples, although the increased depth is far from  
increased accuracy) as well as 44.1 kHz, 48 kHz, 96 kHz, and 192 kHz,  
it becomes even more important to confirm that what we're evaluating  
out of the DAC is actually what we think.

I have certainly had instances where 24-bit FLAC would not play  
correctly on certain pieces of software, but this sort of error was  
immediately evidenced by my metering system.  In that case, I was not  
getting 24-bit samples sent to my DAC, and thus all bets are off when  
trying to evaluate the quality of FLAC.

A proper system will result in FLAC sounding identical to AIFF or  
WAV.  If you hear a difference, then something is wrong with your  
system, not with the FLAC format.

Brian Willoughby
Sound Consulting

___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] 24 bit flac decoding efficiency

[Brian Willoughby]

On Apr 11, 2011, at 15:05, Dennis kerrisk wrote:
 I am trying to decode 192k/24 bit flac on 542 MHZ risk processor,  
 and I am
 having problems. I seem to be right at the edge of what this  
 processor can do.
 So I am wondering what functions would be canidates for assembly code
 optimization? Any suggestions? Or could an older version of code be  
 a batter
 candidate to use?

Does your embedded risc development platform include profiling?  The  
best judge of which function need optimization is you, with the help  
of your development tools.  The profiler should run your code and  
measure the amount of time spent in each function, hopefully  
providing you with a weighted list so that you can start optimizing  
the top candidate right away and then go from there.

Brian Willoughby
Sound Consulting

___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] Support for CAF in flac command-line?

[Brian Willoughby]

On Mar 7, 2011, at 08:46, Paul Davis wrote:
 On Mon, Mar 7, 2011 at 11:25 AM, Brian Willoughby  
 bri...@sounds.wa.com wrote:
 All professional tools use conversion factors such as 0x8000 for
 float-to-int and int-to-float because it has a single significant
 bit, and thus this factor does not increase the bit depth of the
 samples passing through.

 i'm not sure what all means, but i don't think its remotely as clear
 cut as you insist:

http://blog.bjornroche.com/2009/12/int-float-int-its-jungle-out- 
 there.html

 moreover, its critically important to note that this issue arises
 primarily for float=16 bit int conversions, where generally
 dithering should be done anyway. the noise introduced by dithering
 will be at least on the same order of magnitude as the 0.005% error
 caused by different choices in the conversion factor. conversions to
 other bit depths don't face (precisely) the same issue.

 its great that you're absolutely convinced that your view of this is
 right. i generally have a lot of respect for your opinions. the
 problem is that there are 4 other people whose opinions i respect
 equally, and when i've discussed it with them, my only conclusion in
 aggregate has to be it depends.

Thanks.

I should qualify my statements by explaining that I'm talking about  
real music that comes from an A/D, with the understanding that 24-bit  
integers in twos-complement format are the standard.  I should also  
point out that my goal is to preserve the original bits unless an  
operation other than format conversion is specifically enabled.  In  
the context of FLAC - a lossless audio codec - this is entirely  
compatible.

My statements would not be appropriate for synthesized waveforms that  
did not originate in the real world by way of A/D converters.  In  
that world, there is freedom to deviate from the specific limitations  
of A/D converters, and thus there might be some hesitance to apply a  
default conversion that might clip certain synthesized waveforms that  
were created without concern for the realities of analog conversion.   
In that case, I would suggest that those who create such entirely  
synthetic waveforms should take the extra steps necessary to  
condition their signals for actual D/A converters, and then a  
transparent float-to-int conversion would not cause any additional  
problems.  It's only when the synthetic algorithms rely on side  
effects of a lossy conversion that things get hairy.

Brian

___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

[Flac-dev] FLAC__stream_decoder_skip_single_frame() // how is it used?

[Brian Willoughby]

Hello all,

I would like to copy a FLAC file without decoding and then being  
forced to re-encode.  All I really need to do is run the bare minimum  
of the decoder to find FLAC frames and then I should be able to write  
the frames unchanged to the destination file (*).  libFLAC offers the  
FLAC__stream_decoder_skip_single_frame() function to provide half of  
this, but I cannot figure out how to (officially) obtain the encoded  
FLAC frame.  When performing a full decode, the write callback makes  
each frame visible to the application, but that's only after it has  
been decoded, which takes extra CPU time. What I think I need is a  
'skip' callback that just gives the frame header and pointers to the  
encoded data.

Does anyone have any hints?  I'm still looking at the flac command- 
line sources, and also digging into the libFLAC sources to see how  
things work, but I haven't found an obvious solution.  The  
documentation seems to suggest that there are valid reasons to scan  
without decoding, but the only thing that seems possible is to check  
for errors and compute the MD5 sum.  I'd actually like to go further  
than those examples and access the encoded data.

I have a feeling that my question has been asked before, or at least  
a similar topic, but searching my archives back to 2006 doesn't  
reveal anything.

* One potential issue is that the FLAC frame may not be byte-aligned,  
such that perhaps the encoded data can't simply be copied to a new,  
shorter file without shifting bits around.  Then again, perhaps the  
FLAC header is byte-aligned at the start, and only the encoded  
contents are bit-aligned.  I'm sure this particular question is  
answered in the documentation of the file format, but I wanted to  
mention it in case it affects my situation.  I'm certainly planning  
to read the format and API documentation thoroughly.

Brian Willoughby
Sound Consulting

P.S.  My desired application is to take so-called 'hidden' tracks  
from a CD and split them into two files by dropping the silence in  
between.  It's fairly common to have an 'extra' track on a CD that is  
only slightly hidden, where several minutes of silence are inserted  
after the last listed track, followed by the 'hidden' track.  I just  
purchased several lossless music titles online, and a few have these  
hidden tracks.  Of course, I'll preserve the original files in my  
backup, but it would be handy for listening if I could just dump the  
silent sections and keep two shorter tracks - this sure would save  
dead air on my music player.  My idea was that since FLAC already  
encodes silence in a special way (independent, constant.value = 0),  
then it would be far faster and simpler to detect the middle stretch  
of silence by dealing directly with the FLAC-encoded files.  I'm not  
worried about keeping 1 or 2 seconds of silence at either end, as  
would happen when copying whole FLAC frames, but really just want to  
discard most of the long silent sections so I don't have to listen to  
them.

I have code running which can detect the silent sections reliably,  
but only when fully decoding.

___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] Synchronizing a streaming client to the server Was: Idea to possibly improve flac?

[Brian Willoughby]

On Jan 7, 2011, at 20:44, Ben Allison wrote:
 The main problem is in the Ogg layer, in my opinion.

I think you are right.

What is the reason that the Ogg layer was chosen for streaming  
broadcast?  My admittedly naive assumption was that Ogg is a file  
format, not a streaming format.


 Imagine this extreme use-case with __completely made up__ numbers.   
 This
 is a scenario where the server is encoding to FLAC on-the-fly from  
 a raw
 PCM input, either from disk or a live stream.

 Let's say the FLAC block size is 1024 samples, or 23ms at 44100 Hz.
 Let's say each silent block compresses to 1 byte.  Let's also say  
 that the
 Ogg packeting layer wants 4096 bytes before creating a page.  Again -
 these numbers are completely made up, but illustrate the point.  In  
 this
 example, it would take 95 seconds of digital silence before Ogg  
 decided to
 send another page out over the network.

I realize that these are made-up numbers, and I wouldn't expect a  
great deal of precision, but it would seem worthwhile to at least  
look at realistic estimates.

The minimum FRAME blocking in FLAC would be 12 bytes for stereo, not  
just 1 byte.  That's with 40-bit minimum FRAME_HEADER, 8-bit  
SUBFRAME_HEADER, 2 silent CONSTANT 16-bit audio samples, and 16-bit  
FRAME_FOOTER.

This means the 4096-byte Ogg packeting layer would only represent 7.9  
seconds of latency.  I think it's perfectly reasonable for a client  
tuning in to a lossless broadcast to wait just under 8 seconds for  
pre-buffering of the stream.

Better yet, ditch the Ogg packeting layer entirely, and use raw FLAC  
streaming.  That should mean that any size packet is possible.  TCP/ 
IP should allow shorter packets than the maximum, although it may be  
more efficient for broadcast traffic to have packets closer to the  
normal size for file transfers, which would be more like 3.4  
seconds.  I admit that I don't know for sure what restrictions there  
are on internet broadcast packets, but it can't be much worse.

3.4 seconds is a long time to go without a packet, but buffering of  
6.7 seconds or more should easily take care of that.


 If the input audio on the server is coming from a live-source (such as
 simulcast of an FM station) or if the disk I/O is very slow, this  
 can be
 extremely problematic.

 Ben Allison
 Principal Software Engineer
 Nullsoft, Inc.

I don't see why live-source audio would be a problem.  The system  
would simply buffer the required amount of uncompressed audio data  
before calling the FLAC encoder.  The only cost would be latency, and  
all digital broadcast systems that involve codecs involve a serious  
amount of latency.  I can hear a distinct delay between my HDTV set  
tuner and computer USB tuner, since each has a different amount of  
buffering in its pipeline, exacerbated by the digital audio system.

Brian Willoughby
Sound Consulting

___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] Idea to possibly improve flac?

[Brian Willoughby]

First of all, I am not aware of any official source of FLAC files  
that provide MP3 sourced data.  I meticulously check the music I  
purchase, especially when it is 24/48 or 24/96 material, because this  
is new technology, and sometimes people get it wrong.

However, you should be aware that many modern producers use software  
to create their music, and when the software stores sound clips in  
MP3 format, what you end up with is music that sometimes looks like  
MP3.  I recently purchased a second release of an old download from  
an artist who has his material re-mastered.  Since he made such a big  
deal about the re-mastering, I took a close look at the quality.  For  
some reason, the second track looked like an MP3 source, but I'm sure  
it just has to do with the software that was used to create the music  
originally.

In other words, if you try to shut down the FLAC encoder based on an  
FFT, you might have a lot of false triggers!

I purchase a great deal of music, exclusively in FLAC format.  I  
purchase from LINN Records, Bleep.com, Warp Records, and also  
directly from artists like Nine Inch Nails who provide FLAC files.  I  
have never seen anyone provide MP3 quality.

For that matter, OggFLAC seems to be a format that has never been  
used.  Ever.  I have simply never come across a legitimate source of  
music for purchasing which used the OggFLAC format.  I have seen FLAC  
come and go and come back again.

Various online record labels started out with FLAC for bandwidth  
reasons.  Then they seemed to switch over to WAV as bandwidth became  
less of an issue, and I assume that their customers were confused by  
FLAC because of the lack of support in iTunes and other highly  
popular players.  Meanwhile, hardware such as the Sound Devices 700  
Series, the Squeezebox, and many other professional products has  
started with FLAC and stuck with it.  The sites who switched to WAV  
are now bringing back FLAC, but none of them have ever used OggFLAC.

Finally, I think that people who are not embedded firmware developers  
do not understand why the FLAC sources have stopped changing.  What  
we have here is a rare case of a professional set of sources which do  
not have bugs, and which represents a solid standard that does not  
need changing.  People are selling hardware devices in droves, and  
they cannot afford to change their firmware every time some random  
change happens in the FLAC source.  It's actually way better that  
FLAC is not changing.

Even when Apple came out with ALAC, their version of FLAC, I noticed  
that they could not consistently beat FLAC on coding speed and file  
size.  Some audio turns out smaller with ALAC, other audio turns out  
smaller with FLAC.  Overall, the average performance is identical.   
Apple hired some of the most amazing geniuses of physics to design  
ALAC, and if they can't beat the performance of FLAC in all  
situations, then what makes you think there is any reason to make a  
single change to the FLAC sources?

While I'm writing, I also want to respond to the question about how  
to change FLAC so that all of the third party tools pick up the  
change.  Well, I don't think that is possible.  Many tools run the  
command-line flac utility behind the scenes.  Others use the FLAC  
library directly.  The problem is that both of them often run with  
out of date versions of the FLAC code, so no matter which way they  
incorporate the official FLAC sources, you cannot make them update to  
your anti-MP3 version.

On that last note, I want to encourage you to experiment and have fun  
trying to create an MP3 detector that could warn users about quality  
issues.  However, I believe it is extremely unlikely that you would  
ever be successful in getting your code into the official FLAC  
sources.  This kind of change has nothing to do with the official  
FLAC format, and thus I doubt there would be any professional  
interest in changing things just for the sake of change or newness.

Brian Willoughby
Sound Consultinf


On Jan 7, 2011, at 12:56, David Richards wrote:
 Its really sad to hear thats happening but even more sad is the fact
 that flac is becoming a very common format for music on the interweb
 whilst at the same time the development has ceased. I've found some
 severe issues with OggFLAC that essentially make it a useless format
 for streaming, no one cared.

 On Fri, Jan 7, 2011 at 3:42 PM, Jørgen Vigdal jor...@anion.no wrote:
 Due to the fact that more and more users increasingly use MP3   
 320kbps as
 their source for encoding music, and publish it as flac files, I  
 suggest
 that something is done in the flac encoder to possible avoid this.
 My idea is kinda easy/stupid, but might work;
 Implement a function that use a FFT to check if the input has  
 frequencies 
 16kHz, and informs the user that the file would not be encoded  
 unless a
 -force parameter is specified (or at least ask the user if he or  
 she want to
 do

Re: [Flac-dev] Synchronizing a streaming client to the server Was: Idea to possibly improve flac?

[Brian Willoughby]

  
until the two time lines are resynchronized.  But, since this would  
only add or remove silence, there should be absolutely no audible  
glitch.  Perhaps the stream would need more than simple silent flags,  
or resync flags.  It might be necessary to transmit an actual running  
time line counter, with enough bits to count the longest stretch of  
contiguously-clocked audio blocks.  When the broadcast server sees a  
break in the content material, the time code could be reset to zero,  
and this would tell the client to start the sync over, thus avoiding  
dropped samples in the middle of real audio content.


 Anyway what could happen is the client could do a little bit of
 re-sampling here or there to ensure its in sync with the servers
 clock.

 That is how streaming clients usually solve this problem, although  
 is not
 really improving sound quality.

 Its probably not a big deal if you don't resample all the time, just
 when your off by X amount, all of this would just be client side
 preferences. As long as the client side knows its off by X amount
 you could handle it in any number of ways, I'd be fine if its just
 crossfaded to the correct timing if was off by more than half a
 second, then no resampling would ever happen, you would just get a
 weird effect about once an hour, better than a buffer underrun or lag,
 or perhaps the client could look for a half second of silence and just
 cut it out.
I don't think it's a good idea to resample just some of the time,  
although your idea to crossfade would work since it never resamples.   
I think that there are a number of PC-based digital audio playback  
systems, and perhaps even in the television broadcast industry, where  
this idea of intermittent resampling is done.  I hear a regular  
glitch in audio about once per second in many syndicated television  
shows, and my suspicion is that they are speeding up the show so that  
they can sell more commercial time.  Another place that I hear this  
glitching is in some of the PC audio software oriented for DJs which  
can play MP3 files at different speeds and mix them together.  I hear  
the same sound - one glitch per second - and it is very annoying.

But, as you said, a crossfade once per hour would not be as bad.   
Also, the stream could be completely resynchronized even without a  
crossfade.  Some streaming servers are so bad that they can't run for  
hours without rebuffering, but I guess it's probably pretty lazy to  
design something that does that on purpose (the rebuffering, that  
is).  However, as I suggested, it might be better if the broadcast  
server gives hints so that the client player can do these crossfades  
during the silence between tracks.  Using my idea, you'd need to  
crossfade more than once per hour, because there probably isn't  
enough silence to handle it that seldom.  But a fraction of a second  
between tracks several times per hour would never be noticed, unless  
there is a continuous audio broadcast with absolutely no silence.

Brian Willoughby
Sound Consulting

___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] Idea to possibly improve flac?

[Brian Willoughby]

On Jan 7, 2011, at 15:54, Jørgen Vigdal wrote:
 My first suggestion was to use FFT, because I know that 128kbps mp3  
 have a low-pass filter at 16kHz (Fraunhofer IIS Encoder). The  
 program should not decide whether or not the file is a mp3 based on  
 only that, but it could give an indication on that. Perhaps with  
 people not very familiar with advanced audio tools know how to spot  
 out a mp3, they just want to know if the flac they just bought or  
 downloaded is good or not?

What I have found is that audio above 15.8 kHz seems to come and go,  
probably based upon available bits.  While some MP3 may have more of  
a constant state of nothing at those high frequencies, other MP3 bit  
rates will have intermittent content.

What does seem to be true is that no version of MP3 or MP4/AAC will  
encode anything above 20 kHz.  Even with the CD 44.1 kHz sample rate,  
there is still some possibilities for frequencies between 20 kHz and  
22.05 kHz.  Lossy encoding will never encode those frequencies.

With either frequency range, though, you're going to have a slight  
problem detecting absence of frequency information.  That's because  
lossy coding allows noise at all frequencies.  So, your FFT is still  
going to show something above 15.8 kHz, and even something above 20  
kHz, especially if the source was 16-bit instead of 24-bit.  Thus,  
you'll need an intelligent threshold, and perhaps adaptive algorithms  
to detect lossy coding.  It's quite normal for some acoustic  
recordings to have low levels of audio content at higher frequencies,  
especially if the microphone was located a long distance from the  
sound source, because of the high-frequency attenuation properties of  
air given sufficient distance.


 Another test, if the FFT test is unclear, is to check the  
 correlation between left and right channel below 3kHz, which should  
 be just about nothing if the source is a mp3 (since mp3 encoders  
 sum L and R below 3kHz at low bitrates). If also doing further  
 testing, and knowing how the mp3 encoders work, it should be fairly  
 easy to determine if a source file might have been an mp3. At least  
 the program would be able to tell if the input file is poor  
 quality? :)

Some encoders drop information below 10 Hz, some do not.  I am not  
aware of mono coding for 3 kHz and below.  Perhaps this depends upon  
whether the Joint Stereo option is being used or not. In any case, I  
would suggest that very common encoders might not have the properties  
you're describing.


In any case, I don't want to discourage your efforts.  I just think  
it might be harder than you think.  For one thing, lossy coding  
doesn't really drop frequencies as described in most introductory  
texts, but rather what they really do is allow significant  
quantization noise to creep in at different frequencies.  Thus, your  
16-bit CD might be coded at 8-bit or less in some frequencies, where  
it might actually have the equivalent of 24-bit at other  
frequencies.  Since you generally cannot recode lossy files without  
going back to the original, I expect this also means that you can't  
detect everything with an FFT, even if you're running the exact same  
FFT used by the lossy encoder.

It should be a fun learning experience, though!  Keep us posted with  
your findings!

Brian Willoughby
Sound Consulting

___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] Synchronizing a streaming client to the server Was: Idea to possibly improve flac?

[Brian Willoughby]

On Jan 7, 2011, at 17:18, Paul Davis wrote:
 On Fri, Jan 7, 2011 at 7:36 PM, Brian Willoughby  
 bri...@sounds.wa.com wrote:
  I'd like to borrow these ideas, or at
 least similarly-inspired ideas, and have FLAC streaming designed such
 that the stream can tell the playback software when to reset.

 the internals of the slim protocol does this. its all open source. i
 was doing some work on this in 2009 to get it to handle multiple
 sample rates, which it wasn't really designed for, but the reset
 concept is all there.
Awesome!  Can ya help a brotha' out and send some links to the slim  
protocol?  I suppose Google could find it for me, but if you're  
familiar with the project then I'd appreciate an insider's pointers.

A friend has some old slim hardware, and I've been trying to help him  
with converting files for playback, so I do have a little bit of  
experience with the history.

Brian Willoughby
Sound Consulting

___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] Synchronizing a streaming client to the server Was: Idea to possibly improve flac?

[Brian Willoughby]

I just thought of something: Given the maximum supported network  
packet size, and the minimum number of channels (probably stereo) for  
a FLAC broadcast stream, it should be possible to calculate the  
absolute longest time that a single network packet could span.  Once  
you know that time, you could simply double it, and then make sure  
the streaming client always buffers up at least that much time before  
playback is started.  Voila - instant protection against starvation  
due to silent frames being compressed to ultra-tiny packets with a  
long delay.

Some of the comments here have talked about low latency, but I would  
say that low latency has no place in an internet streaming  
broadcast.  I mean, the listened have no frame of reference for  
latency anyway, so what does it matter if the latency is really high?

Now that I think about it this way, I'd say that FLAC and OggFLAC  
should not have any real problems due to compression of silent  
frames.  Any place there is a problem should be blamed on bad  
streaming client / player code, not on the format itself.

Brian Willoughby
Sound Consulting

___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] Synchronizing a streaming client to the server Was: Idea to possibly improve flac?

[Brian Willoughby]

Well, in that light, I suppose it isn't reasonable to expect people  
to wait 23 seconds for their internet streaming broadcast to start  
playing.

Then again, maybe it could be sold as the price for lossless  
streaming!  ;-)

Seriously, though, what about if the Ogg page is not part of the  
picture?  i.e., what about just using pure FLAC and IP?

Brian Willoughby
Sound Consulting


On Jan 7, 2011, at 20:49, David Richards wrote:
 The actual non made up number for 44100 is 23 seconds. :D

 4096 samples, 254 packets in an ogg page.


___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] Cue sheets/Tagging question

[Brian Willoughby]

You should read the manual for the FLAC command line utilities.   
There are many options that would probably achieve what you desire.   
In particular, one of the links that I sent below clearly leads you  
to a page describing how cuesheets are related to the SEEKTABLE, and  
you can create a --seekpoint= marker for each track.

Sometimes you really need to take the time to read everything  
available to you, especially when you have been waiting for months  
for someone else to read it to you.

Brian Willoughby
Sound Consulting


On Nov 11, 2010, at 01:44, Neil Wilkes wrote:
 I still cannot see how to embed this within a long file.
 The documentation talksabout preserving tracklistings from Audio CD  
 - this is *not* what I need.
 I have a series of 24-bit Wave files, not on Audio CD.
 I need to find a way to encode to FLAC and add track points within  
 the stream.
 Is this possible, and how do I do it please?

 I have been trying to find this info for months now, and nobody  
 seems to even know if it is possible.

 Sincerely

 Neil Wilkes

 - Original Message - From: Brian Willoughby  
 bri...@sounds.wa.com
 On Nov 10, 2010, at 12:14, Neil Wilkes wrote:
 Is it possible to place track markers that will be reflected in  
 a  cue sheet within a long FLAC file?
 I have a label who want to offer FLAC downloads of complete  
 albums  - but there have to be track points designated within the  
 FLAC file

 How can I do this please?

 I have not yet started using cue sheets, but I have excerpted what  
 I found in my archive of the FLAC-dev list.  It looks like there  
 are features to preserve a cue sheet.  It may require  
 uncompressing to  WAV, but C. Brown has a player which works  
 directly from the FLAC.

 Brian


 On Sat, 2007-06-23 at 20:55 -0700, Josh Coalson wrote:
 it embeds everything except the tags, see
 http://flac.sourceforge.net/faq.html#general__no_cuesheet_tags
 http://flac.sourceforge.net/  
 documentation_tools_flac.html#flac_options_cuesheet
 http://flac.sourceforge.net/format.html#def_CUESHEET

 Josh


 On Nov 9, 2008, at 06:23, Christopher Brown wrote:
 Hi Helge,

 which players for linux/x86 support flac with embedded cuesheet?
 Is there a list somewhere?

 I don't know of a list, but you can try this one that I wrote:

 http://www.audioplayer51.org

___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] Cue sheets/Tagging question

[Brian Willoughby]

On Nov 10, 2010, at 12:14, Neil Wilkes wrote:
 Is it possible to place track markers that will be reflected in a  
 cue sheet within a long FLAC file?
 I have a label who want to offer FLAC downloads of complete albums  
 - but there have to be track points designated within the FLAC file

 How can I do this please?

I have not yet started using cue sheets, but I have excerpted what I  
found in my archive of the FLAC-dev list.  It looks like there are  
features to preserve a cue sheet.  It may require uncompressing to  
WAV, but C. Brown has a player which works directly from the FLAC.

Brian


On Sat, 2007-06-23 at 20:55 -0700, Josh Coalson wrote:
 it embeds everything except the tags, see
 http://flac.sourceforge.net/faq.html#general__no_cuesheet_tags
 http://flac.sourceforge.net/ 
 documentation_tools_flac.html#flac_options_cuesheet
 http://flac.sourceforge.net/format.html#def_CUESHEET

 Josh


On Nov 9, 2008, at 06:23, Christopher Brown wrote:
 Hi Helge,

 which players for linux/x86 support flac with embedded cuesheet?
 Is there a list somewhere?

 I don't know of a list, but you can try this one that I wrote:

 http://www.audioplayer51.org


___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] Bugfix release?

[Brian Willoughby]

I am not aware of any bugs, which is probably why you do not see a  
steady stream of releases.  I actually find this quite reassuring,  
since I prefer a highly stable and dependable tool as opposed to one  
which is frequently updated.  Every update brings the potential for  
new bugs.

If you have a bug to report, then please mention it.  Otherwise, just  
asking for a new release is probably not something that is productive  
in itself.  I do not intend any offense, I just think that we need a  
good reason to want an update, rather than just creating more work.

P.S.  My main use of FLAC is to archive original recordings of live  
performances.  These are usually 24-bit, 48 kHz recordings, but  
sometimes higher sample rates.  Files are typically AIFF or CAF (the  
latter requires intermediate conversion) and mono, but sometimes I  
will manually combine a stereo pair if my recording system does not  
already allow that.  When a single day's recordings can fill an 8.5  
GB Dual Layer DVD, it becomes important to compress, but lossless is  
an absolute must for original master recordings.  Also, one of my  
recording platforms - the Sound Devices 702 - records directly to  
stereo FLAC, simplifying my archival process.

Brian Willoughby
Sound Consulting


On Jul 22, 2010, at 23:33, grarpamp wrote:
 Hi. It has been some time since I posted to the list, and may be some
 time in the future until I have an oppurtunity to revisit FLAC. So,  
 as I have
 been quite satisfied yet always mindful with FLAC, I simply wanted  
 to encourage
 any potential cumulative bugfix release... and or repository tag/ 
 update
 as such... should it be warranted. It would not go unappreciated by
 the base users, myself included. Thanks :-)

 ps: I cannot speak for any other set of users, but my main current
 use of FLAC is to shrink rips for storage and playback. And to
 expand them back to the identical original rip as needed. Beyond
 that basic requirement, my current interest is in working playback
 plugins. I do not yet need tagging functions as I handle that in the
 filename itself. Nor do I need other functions as of yet.
 As far as I can tell, FLAC has become the leader in lossless.
 Thank you for FLAC!


___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] Compiling in Xcode

[Brian Willoughby]

You're right: any QuickTime component probably requires Xcode.

Based on your comments, the only suggestion I have is to read through  
the compiler options and look for ones which affect the Objective-C++  
mix.  I do recall a few command-line options for changing the  
behavior of the compiler, and they're all available from Xcode if you  
change the build settings.

There are a couple of Apple-hosting mailing lists for Xcode,  
QuickTime, Cocoa, etc.  You might find some help there.

Beyond that, I don't have much to offer at this time.  I worked on a  
similar plugin many years ago, but ran out of time.

Brian Willoughby
Sound Consulting


On Jul 21, 2010, at 06:46, Dmitry Kichenko wrote:
 On 2010-07-21, at 2:24 AM, Brian Willoughby bri...@sounds.wa.com  
 wrote:
 The flac sources are distributed in a configuration that is  
 designed primarily for building from the command-line.  I seem to  
 recall that there might have been an Xcode project in there at  
 some point, but if there was, it never worked well.  In all  
 honesty, I may be getting confused with some other open source  
 project on that last point.

 Thanks for the quick reply!

 I think I couldve been a little more clear in my intentions: I'm  
 trying to compile the Xiph QuickTime component.

 I managed to compile the FLAC.framework as is by changing the SDK  
 over to 10.5 and setting a universal build in the project settings.  
 I haven't tried FLAC++ but at this point I'm mostly interested in  
 getting the Xiph QuickTime component to build, which is what's  
 failing the most it seems.

 Incidentally, I found some reports with similar error messages  
 stemming from intermixing Objective-C and C++ code and gcc getting  
 confused, but since this is an Xcode project I'm wondering how it  
 ever worked, if it has.

 In any event, if you want to compile flac on OSX, then just use  
 Terminal.app

 I'm not sure I can build a QT .component from the terminal without  
 resorting to xcodebuild. The latest source seems to have had  
 changes made to the Xcode project, which gives me hope that at one  
 point, around this time in the summer, one year ago someone was  
 able to successfully get it all to build :)

 Would appreciate any pointers in the direction I should be heading  
 and where to get help.


___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev

Re: [Flac-dev] Compiling in Xcode

[Brian Willoughby]

On Jul 20, 2010, at 22:28, Dmitry Kichenko wrote:
 I'm not sure whether this is the best place to send the question,  
 but since I'm mostly interested in FLAC and there didn't appear to  
 be a better place for it.. here goes.

I think flac-dev is the perfect place for questions about building  
flac.  But the answers are probably going to be geared towards the  
FLAC point of view rather than the Xcode point of view.


 I pulled the latest source code from the SVN repo along with all  
 the external dependencies including FLAC. I've been able to compile  
 the FLAC.framework file using the 10.5 SDK in Xcode 3.2 with a  
 dozen or so warnings, but have been unable to compile the entire  
 Xiph.org QuickTime component project. It seems the dependencies  
 compile just fine, but then things explode with a fountain with  
 hundreds errors all of the same nature, choking seemingly on struct  
 definitions, for the most part:

 src/flac-1.1.0/src/libFLAC/include/protected/file_decoder.h:26:0  
 src/flac-1.1.0/src/libFLAC/include/protected/file_decoder.h:26:  
 error: expected specifier-qualifier-list before  
 'FLAC__FileDecoderState'

 and

 /Users/dmitry/Developer/objc/xiph-qt/build-macosx/../OggImport/src/ 
 importer_types.h:99:0 /Users/dmitry/Developer/objc/xiph-qt/build- 
 macosx/../OggImport/src/importer_types.h:99: error: syntax error  
 before 'Track'

 are some of the repeating examples of errors.

 I've tried building PPC, 32 and x64 builds, as well as the  
 universal ones, but they all fail with the same errors. Would  
 anyone be able to point me in the right direction? It seems like  
 I'm just missing a compiler argument or a project build setting.

The flac sources are distributed in a configuration that is designed  
primarily for building from the command-line.  I seem to recall that  
there might have been an Xcode project in there at some point, but if  
there was, it never worked well.  In all honesty, I may be getting  
confused with some other open source project on that last point.

In any event, if you want to compile flac on OSX, then just use  
Terminal.app

Frankly, if you want to compile flac in Xcode, and you're getting  
mysterious errors, then it's probably a good sign this is a task  
better left to the experts.  It would be great if someone could set  
this up, but it's not necessary since the command line is available.   
I've been using Xcode since it was called ProjectBuilder, and I know  
the environment quite well.  However, I have found that it's usually  
more work than anticipated to convert a command-line build  
environment to a GUI Xcode environment.  It certainly should be  
simple, but it's probably not really worth anyone's time.  That said,  
if anyone does get it working, please share your Xcode project files  
so that Josh can hopefully incorporate them into the source release.

As for my OSX applications which use FLAC, I always add libFLAC.a to  
my project, as compiled and installed by the command-line makefile.   
In other words, rather than compiling the entire source for flac into  
my applications, I simply link to the library. I have not used the  
libFLAC++.a because I prefer not to mix Objective C and C++, but I  
can confirm that the Standard C libFLAC.a works perfectly fine for  
encoding and decoding FLAC files using the standard API.

By the way, I created an Installer.app package which installs the  
entire flac command-line tools and libraries into appropriate  
directories, and then you can just link your OSX applications to the  
precompiled library.  This is the OSX package that is available via  
links at flac.sourceforge.net

Brian Willoughby
Sound Consulting

___
Flac-dev mailing list
Flac-dev@xiph.org
http://lists.xiph.org/mailman/listinfo/flac-dev