Re: [flac-dev] 2GB limit patch
On 3/5/2013 09:57, nu774 wrote: 2. i686-pc-mingw also needs fseeko()/ftello() definitions (only fseeko64() and ftello64() available). Not true for i686-w64-mingw32, it has Linux style LFS64 with _FILE_OFFSET_BITS. signature.asc Description: OpenPGP digital signature ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
Re: [flac-dev] 2GB limit patch
On Mar 4, 2013, at 20:03, nu774 wrote: (2013/03/05 12:27), Cristian Rodríguez wrote: Nothing against you code, that's ugly ..but as it is a first step it can be forgiven :-) I will just force flac to be built with 64 bit file offsets and just reject any caller trying to include/link libflac into a non-lfs program like http://ac-archive.sourceforge.net/largefile/off_t_headers.html Then we can fix all the internal problems, without the need of typedef'ing around ;) Well, I'm afraid typedef'ing or something is mandatory for large file support on win32, since off_t is *always* 32bit on win32 and flac basically uses off_t everywhere for seek offset and file size. I do not see the need to jump to 64-bit merely to get beyond the 2 GB limit. There are modern API that handle 32-bit unsigned file lengths - i.e., up to 4 GB - without going to 64-bit. Besides, RF64 is the proper format for files that exceed 4 GB, while standard RIFF/WAVE should allow reading of files up to 4 GB. For maximum compatibility, writing WAV files over 2 GB should be avoided whenever possible, since so many tools used signed 32-bit file sizes. Brian Willoughby Sound Consulting ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
Re: [flac-dev] 2GB limit patch
On Mar 4, 2013, at 22:50, Erik de Castro Lopo wrote: nu774 wrote: (2013/03/05 12:27), Cristian Rodríguez wrote: Nothing against you code, that's ugly ..but as it is a first step it can be forgiven :-) I will just force flac to be built with 64 bit file offsets and just reject any caller trying to include/link libflac into a non-lfs program like http://ac-archive.sourceforge.net/largefile/off_t_headers.html Then we can fix all the internal problems, without the need of typedef'ing around ;) Well, I'm afraid typedef'ing or something is mandatory for large file support on win32, since off_t is *always* 32bit on win32 and flac basically uses off_t everywhere for seek offset and file size. This requires an API change and hence is something that will need to wait for the next release. This thread started on the topic of WAV files exceeding 2 GB. The FLAC API would only need to change if the FLAC file size limits were altered. Besides, I believe that FLAC is supposed to be a stream with no maximum size limit. No API changes should be necessary for the command-line FLAC to support 4 GB WAV files. Did I miss something? Brian Willoughby Sound Consulting ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
Re: [flac-dev] 2GB limit patch
On 06-03-13 09:43, Brian Willoughby wrote: There is no way for a RIFF/WAVE to exceed 4 GB because all chunks must be enclosed within a global chunk, which is limited to a 32-bit size. I agree it's ugly, but if you take a look at the FLAC bug tracker or the thread on HydrogenAudio that has been discussed, you'll see that it's requested a lot, supporting files 4GB. Apparently no one knew about --ignore-chunk-sizes, because it was not documented, but FLAC already 'supports' it. It displays a warning when you use the option to be 'cautious'. W64 etc. would be much better options, but adoption is still limited AFAIK. ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
Re: [flac-dev] 2GB limit patch
Brian Willoughby wrote: What about fseeko()? I use that in my ObjC library for converting between AIFF, FLAC, and WAVE. It takes an unsigned 32-bit offset, as contrasted with fseek() classic. Unless you need negative offsets, that seems like a simple change. FLAC__stream_decoder_seek_absolute() already takes a FLAC__uint64 offset. I just cast my 32-bit unsigned long and use that in the existing FLAC API. I said offset_t, but I meant off_t. The public header file FLAC/metadata.h returns off_t from funtcion: FLAC_API off_t FLAC__metadata_simple_iterator_get_block_offset(const FLAC__Metadata_SimpleIterator *iterator); That needs to be fixed. Erik -- -- Erik de Castro Lopo http://www.mega-nerd.com/ ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
Re: [flac-dev] 2GB limit patch
(2013/03/06 17:43), Brian Willoughby wrote: That's an awful solution waiting for errors to be introduced into what might otherwise be valid files. There is no way for a RIFF/WAVE to exceed 4 GB because all chunks must be enclosed within a global chunk, which is limited to a 32-bit size. Yeah, but something like --ignore-chunk-sizes is necessary for piped input anyway. I think people should be more or less accustomed to such option. LAME, Nero, opusenc... every popular encoder has such switch (usually named as --ignorelength). It might be an abuse of --ignore-chunk-sizes to use it not only for piped input but also for reading HUGE wave files, but it can be (and has been) used by people that way. Why care about it now? Brian Willoughby Sound Consulting ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
Re: [flac-dev] 2GB limit patch
On 04-03-13 23:19, Erik de Castro Lopo wrote: Link please? http://www.hydrogenaudio.org/forums/index.php?showtopic=99757 Err, thats a link to a post talking about flac's WAV reader being limited to 4Gig files. Problem is, *all* WAV files greater than 4Gig are mal-formed. Due to limitations in the way WAV files are specified, no valid WAV file can ever be over 4Gig. Also, the link in that post contains a flac windows binary which is not of much use. It seems Case hasn't released any source, but the other posts in the thread state what changes have been made to get to this. ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
Re: [flac-dev] 2GB limit patch
Erik de Castro Lopo wrote: Err, thats a link to a post talking about flac's WAV reader being limited to 4Gig files. Problem is, *all* WAV files greater than 4Gig are mal-formed. Due to limitations in the way WAV files are specified, no valid WAV file can ever be over 4Gig. And most don't work over 2 GB. The solution we (Xiph) have used in other projects (opusenc, oggenc) is to treat sizes over ~2 GB as meaning Just keep reading until EOF. See wav_open() audio-in.c in opus-tools. It turns out you need to use slightly less than 2 GB for the limit to handle the output of some tools. We also provide an --ignorelength option to request this behavior regardless of the length in the header (which lets it work with tools that just truncate the size to 32 bits). ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
Re: [flac-dev] 2GB limit patch
This is an older issue reported in 2007: http://lists.xiph.org/pipermail/flac-dev/2007-September/002423.html The fix would be to use _ftelli64 instead of ftell with Visual Studio. http://msdn.microsoft.com/en-us/library/0ys3hc0b%28v=vs.110%29.aspx That's not enough. At least, the followings are also needed. 1. Change off_t to something else. off_t can lead to ABI issue when used in public API, since it's definition can change even on the same system (by setting _FILE_OFFSET_BITS or something), and is always 32bit on mingw, mingw-w64 and MSVC. Since FLAC even takes the trouble of casting seek offset to off_t when calling fseeko(), it gets truncated to 32bit on Win32 even if you use _fseeki64(). This change results in ABI break, but luckily it seems there's only one metadata function that uses off_t in public API. 2. i686-pc-mingw also needs fseeko()/ftello() definitions (only fseeko64() and ftello64() available). 3. stat()/fstat() must be also taken care of, when it is used to get file size. 4. In some place, size of off_t is checked to see if FLAC binary is compiled with large file support. These also need modification. I have made a fork on https://github.com/nu774/FLAC including these patches, but it's not tested enough. Erik already knows at least off_t issue, and he said he'd like to fix it in the future. It needs many lines of change (although it basically is a trivial replacement). ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
Re: [flac-dev] 2GB limit patch
On 03/04/2013 10:57 PM, nu774 wrote: This is an older issue reported in 2007: http://lists.xiph.org/pipermail/flac-dev/2007-September/002423.html The fix would be to use _ftelli64 instead of ftell with Visual Studio. http://msdn.microsoft.com/en-us/library/0ys3hc0b%28v=vs.110%29.aspx That's not enough. At least, the followings are also needed. 1. Change off_t to something else. off_t can lead to ABI issue when used in public API, since it's definition can change even on the same system (by setting _FILE_OFFSET_BITS or something), and is always 32bit on mingw, mingw-w64 and MSVC. Since FLAC even takes the trouble of casting seek offset to off_t when calling fseeko(), it gets truncated to 32bit on Win32 even if you use _fseeki64(). This change results in ABI break, but luckily it seems there's only one metadata function that uses off_t in public API. 2. i686-pc-mingw also needs fseeko()/ftello() definitions (only fseeko64() and ftello64() available). 3. stat()/fstat() must be also taken care of, when it is used to get file size. 4. In some place, size of off_t is checked to see if FLAC binary is compiled with large file support. These also need modification. I have made a fork on https://github.com/nu774/FLAC including these patches, but it's not tested enough. Erik already knows at least off_t issue, and he said he'd like to fix it in the future. It needs many lines of change (although it basically is a trivial replacement). Nothing against you code, that's ugly ..but as it is a first step it can be forgiven :-) I will just force flac to be built with 64 bit file offsets and just reject any caller trying to include/link libflac into a non-lfs program like http://ac-archive.sourceforge.net/largefile/off_t_headers.html Then we can fix all the internal problems, without the need of typedef'ing around ;) ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
Re: [flac-dev] 2GB limit patch
(2013/03/05 12:27), Cristian Rodríguez wrote: Nothing against you code, that's ugly ..but as it is a first step it can be forgiven :-) I will just force flac to be built with 64 bit file offsets and just reject any caller trying to include/link libflac into a non-lfs program like http://ac-archive.sourceforge.net/largefile/off_t_headers.html Then we can fix all the internal problems, without the need of typedef'ing around ;) Well, I'm afraid typedef'ing or something is mandatory for large file support on win32, since off_t is *always* 32bit on win32 and flac basically uses off_t everywhere for seek offset and file size. ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
Re: [flac-dev] 2GB limit patch
nu774 wrote: (2013/03/05 7:19), Erik de Castro Lopo wrote: Err, thats a link to a post talking about flac's WAV reader being limited to 4Gig files. Problem is, *all* WAV files greater than 4Gig are mal-formed. Due to limitations in the way WAV files are specified, no valid WAV file can ever be over 4Gig. That's not an issue. FLAC frontend already has a --ignore-chunk-sizes switch, which exactly does what is needed to handle mal-formed WAV files. Thanks for pointing that out. I've just pushed a couple of patches to add --ignore-chunk-sizes to the --help output and to document them in the HTML docs. Erik -- -- Erik de Castro Lopo http://www.mega-nerd.com/ ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
Re: [flac-dev] 2GB limit patch
nu774 wrote: (2013/03/05 12:27), Cristian Rodríguez wrote: Nothing against you code, that's ugly ..but as it is a first step it can be forgiven :-) I will just force flac to be built with 64 bit file offsets and just reject any caller trying to include/link libflac into a non-lfs program like http://ac-archive.sourceforge.net/largefile/off_t_headers.html Then we can fix all the internal problems, without the need of typedef'ing around ;) Well, I'm afraid typedef'ing or something is mandatory for large file support on win32, since off_t is *always* 32bit on win32 and flac basically uses off_t everywhere for seek offset and file size. This requires an API change and hence is something that will need to wait for the next release. Erik -- -- Erik de Castro Lopo http://www.mega-nerd.com/ ___ flac-dev mailing list flac-dev@xiph.org http://lists.xiph.org/mailman/listinfo/flac-dev
