subject:"\[android\-porting\] Re\: FAT32 file size limited to 2GB"

[android-porting] Re: FAT32 file size limited to 2GB

2010-04-21 Thread Hedge

I have a lot of updates on this thread, but no breakthroughs:
---
I enabled NTFS support in my 2.6.32 kernel build:
  Steps: edit the .config file, added CONFIG_NTFS_FS=y and
CONFIG_NTFS_RW=y
The ADP1 still would not recognize an NTFS-formatted memory card.

I also did more digging into why OpenCORE fails to play back a 2.1GB
file.  In PVMFRecognizerRegistryImpl::CheckForDataAvailability(), it
calls iDataStream-OpenSession().  iDataStream is a
PVMIDataStreamSyncInterfaceRefImpl object.
Internally it creates a new iFileObject [result == 0].
Then it seeks to the end of the file to try to determine file size
[iFileNumBytes = (TOsclFileOffsetInt32)iFileObject-Tell()] before
trying to reset the file position back to the beginning.

Next, PVMFRecognizerRegistryImpl::CheckForDataAvailability() calls
iDataStream-QueryReadCapacity().
This calls GetCurrentPointerPosition(), which calls iFileObject-
Tell(), which returns -1 and causes a Leave 103.  It looks like an I/
O limitation.
---
Other things I've discovered:  On 
http://opencore.net/files/OpenCORE_roadmap.pdf,
PV plans to add large file support to MP4 in May of 2010.  In my
Donut_plus_aosp build, I see OSCL_HAS_LARGE_FILE_SUPPORT = 1 in
osclconfig_io.h.  Does anyone know what a large file is by PV's
standards?
---

Thanks for any ideas,
~Andrew

On Mar 14, 10:07 pm, Hedge awoo...@gmail.com wrote:
 Specifically I'm trying to playback files in OpenCORE, not with an NDK
 app, but as a test app it might be useful.  Using NTFS sounds like it
 might pan out too.
 Thanks for the tips everyone, and I'll try them out when I get back in
 the office.

 On Mar 13, 6:48 pm, Dianne Hackborn hack...@android.com wrote:

  On Fri, Mar 12, 2010 at 4:50 PM, Hedge awoo...@gmail.com wrote:
   Has anyone heard if this will be fixed in Android 3.0?

  I haven't even heard of Android 3.0.

  --
  Dianne Hackborn
  Android framework engineer
  hack...@android.com

  Note: please don't send private questions to me, as I don't have time to
  provide private support, and so won't reply to such e-mails.  All such
  questions should be posted on public forums, where I and others can see and
  answer them.



-- 
unsubscribe: android-porting+unsubscr...@googlegroups.com
website: http://groups.google.com/group/android-porting

Re: [android-porting] Re: FAT32 file size limited to 2GB

2010-03-13 Thread hedwin

NTFS doesn't have this limitation. Adding this to the kernel is not a big
deal.
Also NTFS FS is abstracted by the kernel so bionic should not care what is
below
the bonnet. What needs to be checked is the mount command whether this for
some reason would not allow NTFS mounts.

On Sat, Mar 13, 2010 at 1:50 AM, Hedge awoo...@gmail.com wrote:

No, I'm not sure how to add NTFS to the kernel. If the 2GB
limitations apply to the Linux kernel or bionic libraries, would NTFS
even help? I'd prefer to add 4GB capabilities to FAT32 if possible.

As far as the FAT32 2GB limits, I see _FILE_OFFSET_BITS == 64 in some
files (http://www.gnu.org/software/libc/manual/html_node/Opening-
Streams.html#index-fopen64-931http://www.gnu.org/software/libc/manual/html_node/Opening-%0AStreams.html#index-fopen64-931).
I wrote test programs for low-level
file access in Java and C and I can open large files there. However,
seeks fail beyond the 2GB limit, probably because the offset variable
(off_t) is a 4-byte long.

Has anyone heard if this will be fixed in Android 3.0?

On Mar 9, 5:07 pm, hedwin hedwin.kon...@gmail.com wrote:
Did you configure NTFS in your kernel?. Think it is disabled per default
(only checked android-x86).

On Tue, Mar 9, 2010 at 11:31 PM, Hedge awoo...@gmail.com wrote:
Thanks everyone for your help so far.

A few more things:
* FAT32 is not limited to 2GB:
http://en.wikipedia.org/wiki/File_Allocation_Table#FAT32
(The maximum possible size for a file on a FAT32 volume is 4 GB minus
1 byte)
* I did try formatting an SDHC card as NTFS but unfortunately the
Android device would not recognize it. Also, ext3 wouldn't work on
most Windows users' machines without downloading additional programs.
* I'm surprised this issue hasn't caused more of a stir before. I only
found one other mention of it (http://groups.google.com/group/android-
platform/browse_thread/thread/7c375594e5cc5427/2f397fe2eb829bae?
lnk=gstq=file+size+limit#2f397fe2eb829bae
http://groups.google.com/group/android-%0Aplatform/browse_thread/thre...
)

On Mar 9, 2:24 pm, Hedge awoo...@gmail.com wrote:
I found this bit of info about a 2GB limit in Linux kernels before
v2.4.0 (http://linuxmafia.com/faq/VALinux-kb/2gb-filesize-
limit.html). But Android appears to be using v2.6 (http://
en.wikipedia.org/wiki/Android_%28operating_system
%29#Open_Handset_Alliance).

On Mar 9, 1:37 pm, Deva R r.deva...@gmail.com wrote:

The underlying file i/o c library calls are likely to be the
common
32 bit

signed versions unless someone specifically chose the 64 bit
versions.
nope.. may be you meant FAT16. fat32 max file size is 232-1 bytes
(~4GB).

it looks odd why access beyond 2G should fail. As chris said below,
there
might be a break in i/o lib.

On Wed, Mar 10, 2010 at 2:52 AM, hedwin hedwin.kon...@gmail.com
wrote:
FAT32 itself is limited to 2GB. If you need to handle files
larger
than 2GB
you either need to use NTFS or ext3 or higher.

On Tue, Mar 9, 2010 at 4:12 AM, Chris Stratton
cs07...@gmail.com
wrote:

The underlying file i/o c library calls are likely to be the
common
32
bit signed versions unless someone specifically chose the 64 bit
versions.

On Mar 8, 1:13 pm, Hedge awoo...@gmail.com wrote:
I am trying to play back 3GB videos from an SDHC card in
OpenCORE
v2.05 in Cupcake. The player crashes every time. ADB shell
reports
the
file size to be a negative number (overflow).

I believe the maximum file size on FAT32 is 4GB (2^32 - 1).
The
maximum size I can access on Android is actually 2GB (2^31 -
1),
which
makes me think the addressing is performed with a signed
integer
instead of unsigned.

Does anyone know how to access files that are larger than 2GB?
Was
this fixed in a later version of Android? Can I change the
type of
some kernel variable to an unsigned int to unlock the extra
addressing
space?

--
unsubscribe:
android-porting+unsubscr...@googlegroups.comandroid-porting%2bunsubscr...@googlegroups.com
android-porting%2bunsubscr...@googlegroups.comandroid-porting%252bunsubscr...@googlegroups.com

android-porting%2bunsubscr...@googlegroups.comandroid-porting%252bunsubscr...@googlegroups.com
android-porting%252bunsubscr...@googlegroups.comandroid-porting%25252bunsubscr...@googlegroups.com

website:http://groups.google.com/group/android-porting

android-porting%2bunsubscr...@googlegroups.comandroid-porting%252bunsubscr...@googlegroups.com

[android-porting] Re: FAT32 file size limited to 2GB

2010-03-13 Thread Chris Stratton

Try writing something in the ndk (or a command line executable if you
have the toolchain wrapper scripts) explicitly using 64 bit file io.

And compile and test it against a fat32 volume on a normal linux box
first.

Hedge wrote:
Thanks everyone for your help so far.

On Mar 9, 1:37 pm, Deva R r.deva...@gmail.com wrote:

The underlying file i/o c library calls are likely to be the common 32
bit

signed versions unless someone specifically chose the 64 bit versions.
nope.. may be you meant FAT16. fat32 max file size is 232-1 bytes (~4GB).

it looks odd why access beyond 2G should fail. As chris said below, there
might be a break in i/o lib.

On Wed, Mar 10, 2010 at 2:52 AM, hedwin hedwin.kon...@gmail.com wrote:
FAT32 itself is limited to 2GB. If you need to handle files larger than
2GB
you either need to use NTFS or ext3 or higher.

On Tue, Mar 9, 2010 at 4:12 AM, Chris Stratton cs07...@gmail.com
wrote:

The underlying file i/o c library calls are likely to be the common 32
bit signed versions unless someone specifically chose the 64 bit
versions.

On Mar 8, 1:13 pm, Hedge awoo...@gmail.com wrote:
I am trying to play back 3GB videos from an SDHC card in OpenCORE
v2.05 in Cupcake. The player crashes every time. ADB shell reports
the
file size to be a negative number (overflow).

I believe the maximum file size on FAT32 is 4GB (2^32 - 1). The
maximum size I can access on Android is actually 2GB (2^31 - 1),
which
makes me think the addressing is performed with a signed integer
instead of unsigned.

Does anyone know how to access files that are larger than 2GB? Was
this fixed in a later version of Android? Can I change the type of
some kernel variable to an unsigned int to unlock the extra
addressing
space?

--
unsubscribe:
android-porting+unsubscr...@googlegroups.comandroid-porting%2bunsubscr...@googlegroups.com
website:http://groups.google.com/group/android-porting

--
unsubscribe: android-porting+unsubscr...@googlegroups.com
website: http://groups.google.com/group/android-porting

Re: [android-porting] Re: FAT32 file size limited to 2GB

2010-03-13 Thread Dianne Hackborn

On Fri, Mar 12, 2010 at 4:50 PM, Hedge awoo...@gmail.com wrote:

 Has anyone heard if this will be fixed in Android 3.0?


I haven't even heard of Android 3.0.

-- 
Dianne Hackborn
Android framework engineer
hack...@android.com

Note: please don't send private questions to me, as I don't have time to
provide private support, and so won't reply to such e-mails.  All such
questions should be posted on public forums, where I and others can see and
answer them.

-- 
unsubscribe: android-porting+unsubscr...@googlegroups.com
website: http://groups.google.com/group/android-porting

[android-porting] Re: FAT32 file size limited to 2GB

2010-03-12 Thread Hedge

No, I'm not sure how to add NTFS to the kernel. If the 2GB
limitations apply to the Linux kernel or bionic libraries, would NTFS
even help? I'd prefer to add 4GB capabilities to FAT32 if possible.

As far as the FAT32 2GB limits, I see _FILE_OFFSET_BITS == 64 in some
files (http://www.gnu.org/software/libc/manual/html_node/Opening-
Streams.html#index-fopen64-931). I wrote test programs for low-level
file access in Java and C and I can open large files there. However,
seeks fail beyond the 2GB limit, probably because the offset variable
(off_t) is a 4-byte long.

Has anyone heard if this will be fixed in Android 3.0?

On Mar 9, 5:07 pm, hedwin hedwin.kon...@gmail.com wrote:
Did you configure NTFS in your kernel?. Think it is disabled per default
(only checked android-x86).

On Tue, Mar 9, 2010 at 11:31 PM, Hedge awoo...@gmail.com wrote:
Thanks everyone for your help so far.

A few more things:
* FAT32 is not limited to 2GB:
http://en.wikipedia.org/wiki/File_Allocation_Table#FAT32
(The maximum possible size for a file on a FAT32 volume is 4 GB minus
1 byte)
* I did try formatting an SDHC card as NTFS but unfortunately the
Android device would not recognize it. Also, ext3 wouldn't work on
most Windows users' machines without downloading additional programs.
* I'm surprised this issue hasn't caused more of a stir before. I only
found one other mention of it (http://groups.google.com/group/android-
platform/browse_thread/thread/7c375594e5cc5427/2f397fe2eb829bae?
lnk=gstq=file+size+limit#2f397fe2eb829baehttp://groups.google.com/group/android-%0Aplatform/browse_thread/thre...
)

On Mar 9, 1:37 pm, Deva R r.deva...@gmail.com wrote:

The underlying file i/o c library calls are likely to be the common
32 bit

signed versions unless someone specifically chose the 64 bit versions.
nope.. may be you meant FAT16. fat32 max file size is 232-1 bytes
(~4GB).

it looks odd why access beyond 2G should fail. As chris said below,
there
might be a break in i/o lib.

On Wed, Mar 10, 2010 at 2:52 AM, hedwin hedwin.kon...@gmail.com
wrote:
FAT32 itself is limited to 2GB. If you need to handle files larger
than 2GB
you either need to use NTFS or ext3 or higher.

On Tue, Mar 9, 2010 at 4:12 AM, Chris Stratton cs07...@gmail.com
wrote:

The underlying file i/o c library calls are likely to be the common
32
bit signed versions unless someone specifically chose the 64 bit
versions.

On Mar 8, 1:13 pm, Hedge awoo...@gmail.com wrote:
I am trying to play back 3GB videos from an SDHC card in OpenCORE
v2.05 in Cupcake. The player crashes every time. ADB shell reports
the
file size to be a negative number (overflow).

I believe the maximum file size on FAT32 is 4GB (2^32 - 1). The
maximum size I can access on Android is actually 2GB (2^31 - 1),
which
makes me think the addressing is performed with a signed integer
instead of unsigned.

Does anyone know how to access files that are larger than 2GB? Was
this fixed in a later version of Android? Can I change the type of
some kernel variable to an unsigned int to unlock the extra
addressing
space?

website:http://groups.google.com/group/android-porting

--
unsubscribe:
android-porting+unsubscr...@googlegroups.comandroid-porting%2bunsubscr...@googlegroups.com
website:http://groups.google.com/group/android-porting

--
unsubscribe: android-porting+unsubscr...@googlegroups.com
website: http://groups.google.com/group/android-porting

Re: [android-porting] Re: FAT32 file size limited to 2GB

2010-03-09 Thread hedwin

FAT32 itself is limited to 2GB. If you need to handle files larger than 2GB
you either need to use NTFS or ext3 or higher.



On Tue, Mar 9, 2010 at 4:12 AM, Chris Stratton cs07...@gmail.com wrote:

 The underlying file i/o c library calls are likely to be the common 32
 bit signed versions unless someone specifically chose the 64 bit
 versions.

 On Mar 8, 1:13 pm, Hedge awoo...@gmail.com wrote:
  I am trying to play back 3GB videos from an SDHC card in OpenCORE
  v2.05 in Cupcake. The player crashes every time. ADB shell reports the
  file size to be a negative number (overflow).
 
  I believe the maximum file size on FAT32 is 4GB (2^32 - 1). The
  maximum size I can access on Android is actually 2GB (2^31 - 1), which
  makes me think the addressing is performed with a signed integer
  instead of unsigned.
 
  Does anyone know how to access files that are larger than 2GB? Was
  this fixed in a later version of Android? Can I change the type of
  some kernel variable to an unsigned int to unlock the extra addressing
  space?

 --
 unsubscribe: 
 android-porting+unsubscr...@googlegroups.comandroid-porting%2bunsubscr...@googlegroups.com
 website: http://groups.google.com/group/android-porting


-- 
unsubscribe: android-porting+unsubscr...@googlegroups.com
website: http://groups.google.com/group/android-porting

Re: [android-porting] Re: FAT32 file size limited to 2GB

2010-03-09 Thread Deva R

The underlying file i/o c library calls are likely to be the common 32 bit
signed versions unless someone specifically chose the 64 bit versions.
nope.. may be you meant FAT16. fat32 max file size is 232-1 bytes (~4GB).

it looks odd why access beyond 2G should fail. As chris said below, there
might be a break in i/o lib.

On Wed, Mar 10, 2010 at 2:52 AM, hedwin hedwin.kon...@gmail.com wrote:

FAT32 itself is limited to 2GB. If you need to handle files larger than 2GB
you either need to use NTFS or ext3 or higher.

On Tue, Mar 9, 2010 at 4:12 AM, Chris Stratton cs07...@gmail.com wrote:

The underlying file i/o c library calls are likely to be the common 32
bit signed versions unless someone specifically chose the 64 bit
versions.

On Mar 8, 1:13 pm, Hedge awoo...@gmail.com wrote:
I am trying to play back 3GB videos from an SDHC card in OpenCORE
v2.05 in Cupcake. The player crashes every time. ADB shell reports the
file size to be a negative number (overflow).

I believe the maximum file size on FAT32 is 4GB (2^32 - 1). The
maximum size I can access on Android is actually 2GB (2^31 - 1), which
makes me think the addressing is performed with a signed integer
instead of unsigned.

Does anyone know how to access files that are larger than 2GB? Was
this fixed in a later version of Android? Can I change the type of
some kernel variable to an unsigned int to unlock the extra addressing
space?

--
unsubscribe:
android-porting+unsubscr...@googlegroups.comandroid-porting%2bunsubscr...@googlegroups.com
website: http://groups.google.com/group/android-porting

--
unsubscribe: android-porting+unsubscr...@googlegroups.com
website: http://groups.google.com/group/android-porting

[android-porting] Re: FAT32 file size limited to 2GB

2010-03-09 Thread Hedge

I found this bit of info about a 2GB limit in Linux kernels before
v2.4.0 (http://linuxmafia.com/faq/VALinux-kb/2gb-filesize-
limit.html). But Android appears to be using v2.6 (http://
en.wikipedia.org/wiki/Android_%28operating_system
%29#Open_Handset_Alliance).

On Mar 9, 1:37 pm, Deva R r.deva...@gmail.com wrote:
The underlying file i/o c library calls are likely to be the common 32 bit

signed versions unless someone specifically chose the 64 bit versions.
nope.. may be you meant FAT16. fat32 max file size is 232-1 bytes (~4GB).

it looks odd why access beyond 2G should fail. As chris said below, there
might be a break in i/o lib.

On Wed, Mar 10, 2010 at 2:52 AM, hedwin hedwin.kon...@gmail.com wrote:
FAT32 itself is limited to 2GB. If you need to handle files larger than 2GB
you either need to use NTFS or ext3 or higher.

On Tue, Mar 9, 2010 at 4:12 AM, Chris Stratton cs07...@gmail.com wrote:

The underlying file i/o c library calls are likely to be the common 32
bit signed versions unless someone specifically chose the 64 bit
versions.

On Mar 8, 1:13 pm, Hedge awoo...@gmail.com wrote:
I am trying to play back 3GB videos from an SDHC card in OpenCORE
v2.05 in Cupcake. The player crashes every time. ADB shell reports the
file size to be a negative number (overflow).

I believe the maximum file size on FAT32 is 4GB (2^32 - 1). The
maximum size I can access on Android is actually 2GB (2^31 - 1), which
makes me think the addressing is performed with a signed integer
instead of unsigned.

Does anyone know how to access files that are larger than 2GB? Was
this fixed in a later version of Android? Can I change the type of
some kernel variable to an unsigned int to unlock the extra addressing
space?

--
unsubscribe:
android-porting+unsubscr...@googlegroups.comandroid-porting%2bunsubscr...@googlegroups.com
website:http://groups.google.com/group/android-porting

--
unsubscribe: android-porting+unsubscr...@googlegroups.com
website: http://groups.google.com/group/android-porting

[android-porting] Re: FAT32 file size limited to 2GB

2010-03-09 Thread Hedge

Thanks everyone for your help so far.

On Mar 9, 1:37 pm, Deva R r.deva...@gmail.com wrote:

The underlying file i/o c library calls are likely to be the common 32 bit