Bug #27792 [Com]: [PATCH] Functions fail on large files (filesize,is_file,is_dir,readdir)
Edit report at https://bugs.php.net/bug.php?id=27792edit=1 ID: 27792 Comment by: mail dot pourri at laposte dot net Reported by:kode at kodekrash dot com Summary:[PATCH] Functions fail on large files (filesize,is_file,is_dir,readdir) Status: Critical Type: Bug Package:Filesystem function related Operating System: * (LFS) PHP Version:5.*, 6CVS (2009-04-30) Block user comment: N Private report: N New Comment: This is the most absurd comment I've ever seen! The software does not work as expected, please fix your machine. I've attached a new updated version of the patch against version 5.5.1 I've no time to backport the patch to any previous version, please be my guest if you need it. Some comments about the patch: 1) Changes size_t to off_t at required places in the code (this will only work if the LFS mode is specified at compilation/configuration time, that is if you type CFLAGS='-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64' ./configure --usual-args 2) Pass all the tests that where passing without the patch 3) Fix a test that was failing without the patch (you can't mmap a pipe's file descriptor, the patch prevent doing so instead of failing at runtime) 4) Returns a double when/where the file size/offset is larger than 2^32 bytes. This means that file with size up to 2^52 (the size of a double's mantissa) can now be used with PHP on 32 bits machine. This is not the plain 64 bits, but adding a new type to PHP for 64 bits integer would be a much larger work. For file size/offset below 2^32 bytes, the previous type (integer) is still used, so backward compatibility is kept. For file larger than 2^52 bytes, approximate values are used (this is still not an issue since reading/writing such file would still work since it's done in C) 5) Loops on copy_stream/passthru until the complete stream is read, unlike the previous behavior of only reading up to the first 2GB of the file. Let me know if it helps. Previous Comments: [2013-05-29 09:57:27] Sjon at hortensius dot net According to http://3v4l.org/sBClC; this is not a problem on a 64 bits linux machine (on any php version). I propose (since this bug obviously won't be fixed) to close this issue and tell people to use 64 bits machines instead of trying to get LFS support in PHP [2012-08-09 13:28:31] adu at rdsor dot ro @marcb I tested it in PHP 5.4.0 (cli) (built: Apr 12 2012 13:02:59)) on Ubuntu 12.04 kernel 3.0.0-19 i686 and the BUG is still here. Reproduced code: $fname = 'file_of_7GB'; var_dump(filesize( $fname )); // PHP Warning: filesize(): stat failed for file_of_7GB in php_bug_27792.php on line 4 // dumps bool(false) var_dump(is_file( $fname )); // dumps bool(false) var_dump(is_dir( $fname )); // dumps bool(false) $fname = 'file_of_354MB'; var_dump(filesize( $fname )); // dumps int(370336155) var_dump(is_file( $fname )); // dumps bool(true) var_dump(is_dir( $fname )); // dumps bool(false) [2011-01-05 04:46:23] marcb at voicemeup dot com Is anyone able to confirm if this was fixed in any of the 5.X branch ? This is a pretty stupid problem to have to deal with ! [2009-11-12 10:27:45] boite dot pour dot spam at gmail dot com The patch from Wez doesn't work, as it assumes size_t are 64 bits, which is not the case, even when LFS is defined. The patch from Mail Pourri works on 5.3.0 (I haven't tested in 5.3.1) [2009-09-14 08:59:36] j...@php.net The latest patch for this: http://www.php.net/~wez/lfs.diff The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=27792 -- Edit this bug report at https://bugs.php.net/bug.php?id=27792edit=1
Req #48886 [Com]: [PATCH] Add support for 64bits file handling on 32 bits architecture
Edit report at https://bugs.php.net/bug.php?id=48886edit=1
ID: 48886
Comment by: mail dot pourri at laposte dot net
Reported by:mail dot pourri at laposte dot net
Summary:[PATCH] Add support for 64bits file handling on 32
bits architecture
Status: Not a bug
Type: Feature/Change Request
Package:Feature/Change Request
Operating System: All
PHP Version:5.3.0
Block user comment: N
Private report: N
New Comment:
Please check the attached patch against PHP 5.5.1.
I've no time to backport it to all PHP version.
It pass all tests that where passing without the patch.
Configure with CFLAGS='-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
Hope it helps
Previous Comments:
[2012-01-28 19:20:05] luigi dot assom at gmail dot com
Could you please instruct how to add the patch?
I don't know if this is what i need...
I am trying to work on wikipedia dump
i am using xampp and bigdump.php
Importing fails with:
UNEXPECTED: Can't set file pointer behind the end of file
I thought it is because of this patch..
I am not a programmer..
using mac..
please could you tell me what to do?
thank you so much,
Luigi
[2009-11-28 21:12:16] j...@php.net
1. You should provide patch against all branches (PHP_5_2, PHP_5_3 and HEAD)
2. Put it online somewhere where it can be downloaded. Adding them here is
quite useless since they can not be applied.
3. Check also bug #27792 of which this is a duplicate of.
4. Mail intern...@lists.php.net about it.
[2009-09-11 15:25:49] boite dot pour dot spam at gmail dot com
It's working. You should talk about this on the dev mailing list
[2009-07-11 13:39:03] mail dot pourri at laposte dot net
diff -aur php-5.3.0.orig/ext/phar/phar_internal.h
php-5.3.0/ext/phar/phar_internal.h
--- php-5.3.0.orig/ext/phar/phar_internal.h 2009-05-13 22:25:43.0
+0200
+++ php-5.3.0/ext/phar/phar_internal.h 2009-07-11 15:10:04.0 +0200
@@ -534,8 +534,15 @@
return FAILURE;
}
#else
-# define phar_stream_copy_to_stream(src, dest, maxlen, len)
_php_stream_copy_to_stream_ex((src), (dest), (maxlen), (len) STREAMS_CC
TSRMLS_CC)
-
+static inline size_t phar_stream_copy_to_stream(php_stream *src, php_stream
*dest, size_t maxlen, size_t *len)
+{
+ off_t _maxlen = maxlen == (size_t)PHP_STREAM_COPY_ALL ?
PHP_STREAM_COPY_ALL : maxlen, _len = 0;
+ size_t ret = php_stream_copy_to_stream_ex(src, dest, _maxlen, _len);
+ if (ret == SUCCESS) {
+ if (len) *len = (size_t)_len;
+ } else if (len) *len = 0;
+ return ret;
+}
#endif
#if PHP_VERSION_ID = 6
diff -aur php-5.3.0.orig/ext/standard/file.c php-5.3.0/ext/standard/file.c
--- php-5.3.0.orig/ext/standard/file.c 2009-06-22 13:37:30.0 +0200
+++ php-5.3.0/ext/standard/file.c 2009-07-11 15:23:32.0 +0200
@@ -594,7 +594,7 @@
char *filename;
int filename_len;
zval *data;
- int numbytes = 0;
+ off_t numbytes = 0;
long flags = 0;
zval *zcontext = NULL;
php_stream_context *context = NULL;
@@ -642,7 +642,7 @@
switch (Z_TYPE_P(data)) {
case IS_RESOURCE: {
- size_t len;
+ off_t len;
if (php_stream_copy_to_stream_ex(srcstream, stream,
PHP_STREAM_COPY_ALL, len) != SUCCESS) {
numbytes = -1;
} else {
@@ -659,9 +659,9 @@
case IS_STRING:
if (Z_STRLEN_P(data)) {
- numbytes = php_stream_write(stream,
Z_STRVAL_P(data), Z_STRLEN_P(data));
- if (numbytes != Z_STRLEN_P(data)) {
- php_error_docref(NULL TSRMLS_CC,
E_WARNING, Only %d of %d bytes written, possibly out of free disk space,
numbytes, Z_STRLEN_P(data));
+ numbytes = (off_t)php_stream_write(stream,
Z_STRVAL_P(data), Z_STRLEN_P(data));
+ if (numbytes != (off_t)Z_STRLEN_P(data)) {
+ php_error_docref(NULL TSRMLS_CC,
E_WARNING, Only %d of %d bytes written, possibly out of free disk space,
(size_t)numbytes, Z_STRLEN_P(data));
numbytes = -1;
}
}
@@ -680,13 +680,13 @@
convert_to_string(*tmp);
}
if (Z_STRLEN_PP(tmp
#48886 [NEW]: [PATCH] Add support for 64bits file handling on 32 bits architecture
From: mail dot pourri at laposte dot net Operating system: All PHP version: 5.3.0 PHP Bug Type: Feature/Change Request Bug description: [PATCH] Add support for 64bits file handling on 32 bits architecture Description: As currently all integers are 32 bits on 32 bits processor, even when php is compiled with Large File Support, all the functions that use integer instead of a right format (stat, filesize, ftell, etc...) don't report the right size when a file is 4GB. The attached patch solve the issue by using doubles instead of int when a size overflow 32 bits (so as IEEE754 specifies, we have 53 bits of real, integer, value in a double). Unlike Wes patch posted earlier, that didn't work, I've modified all the code that still used size_t to use off_t, modifying the extensions too, memory mapping (that had a bug as it previously failed when offset used, BTW), etc... It works when applyied to released 5.3.0 with patch -p1. -- Edit bug report at http://bugs.php.net/?id=48886edit=1 -- Try a CVS snapshot (PHP 5.2): http://bugs.php.net/fix.php?id=48886r=trysnapshot52 Try a CVS snapshot (PHP 5.3): http://bugs.php.net/fix.php?id=48886r=trysnapshot53 Try a CVS snapshot (PHP 6.0): http://bugs.php.net/fix.php?id=48886r=trysnapshot60 Fixed in CVS: http://bugs.php.net/fix.php?id=48886r=fixedcvs Fixed in CVS and need be documented: http://bugs.php.net/fix.php?id=48886r=needdocs Fixed in release: http://bugs.php.net/fix.php?id=48886r=alreadyfixed Need backtrace: http://bugs.php.net/fix.php?id=48886r=needtrace Need Reproduce Script: http://bugs.php.net/fix.php?id=48886r=needscript Try newer version: http://bugs.php.net/fix.php?id=48886r=oldversion Not developer issue: http://bugs.php.net/fix.php?id=48886r=support Expected behavior: http://bugs.php.net/fix.php?id=48886r=notwrong Not enough info: http://bugs.php.net/fix.php?id=48886r=notenoughinfo Submitted twice: http://bugs.php.net/fix.php?id=48886r=submittedtwice register_globals: http://bugs.php.net/fix.php?id=48886r=globals PHP 4 support discontinued: http://bugs.php.net/fix.php?id=48886r=php4 Daylight Savings:http://bugs.php.net/fix.php?id=48886r=dst IIS Stability: http://bugs.php.net/fix.php?id=48886r=isapi Install GNU Sed: http://bugs.php.net/fix.php?id=48886r=gnused Floating point limitations: http://bugs.php.net/fix.php?id=48886r=float No Zend Extensions: http://bugs.php.net/fix.php?id=48886r=nozend MySQL Configuration Error: http://bugs.php.net/fix.php?id=48886r=mysqlcfg
#48886 [Com]: [PATCH] Add support for 64bits file handling on 32 bits architecture
ID: 48886
Comment by: mail dot pourri at laposte dot net
Reported By: mail dot pourri at laposte dot net
Status: Open
Bug Type: Feature/Change Request
Operating System: All
PHP Version: 5.3.0
New Comment:
diff -aur php-5.3.0.orig/ext/phar/phar_internal.h
php-5.3.0/ext/phar/phar_internal.h
--- php-5.3.0.orig/ext/phar/phar_internal.h 2009-05-13
22:25:43.0 +0200
+++ php-5.3.0/ext/phar/phar_internal.h 2009-07-11 15:10:04.0
+0200
@@ -534,8 +534,15 @@
return FAILURE;
}
#else
-# define phar_stream_copy_to_stream(src, dest, maxlen,
len)_php_stream_copy_to_stream_ex((src), (dest), (maxlen), (len)
STREAMS_CC TSRMLS_CC)
-
+static inline size_t phar_stream_copy_to_stream(php_stream *src,
php_stream *dest, size_t maxlen, size_t *len)
+{
+ off_t _maxlen = maxlen == (size_t)PHP_STREAM_COPY_ALL ?
PHP_STREAM_COPY_ALL : maxlen, _len = 0;
+ size_t ret = php_stream_copy_to_stream_ex(src, dest, _maxlen,
_len);
+ if (ret == SUCCESS) {
+ if (len) *len = (size_t)_len;
+ } else if (len) *len = 0;
+ return ret;
+}
#endif
#if PHP_VERSION_ID = 6
diff -aur php-5.3.0.orig/ext/standard/file.c
php-5.3.0/ext/standard/file.c
--- php-5.3.0.orig/ext/standard/file.c 2009-06-22 13:37:30.0
+0200
+++ php-5.3.0/ext/standard/file.c 2009-07-11 15:23:32.0 +0200
@@ -594,7 +594,7 @@
char *filename;
int filename_len;
zval *data;
- int numbytes = 0;
+ off_t numbytes = 0;
long flags = 0;
zval *zcontext = NULL;
php_stream_context *context = NULL;
@@ -642,7 +642,7 @@
switch (Z_TYPE_P(data)) {
case IS_RESOURCE: {
- size_t len;
+ off_t len;
if (php_stream_copy_to_stream_ex(srcstream, stream,
PHP_STREAM_COPY_ALL, len) != SUCCESS) {
numbytes = -1;
} else {
@@ -659,9 +659,9 @@
case IS_STRING:
if (Z_STRLEN_P(data)) {
- numbytes = php_stream_write(stream,
Z_STRVAL_P(data),
Z_STRLEN_P(data));
- if (numbytes != Z_STRLEN_P(data)) {
- php_error_docref(NULL TSRMLS_CC,
E_WARNING, Only %d of %d bytes
written, possibly out of free disk space, numbytes, Z_STRLEN_P(data));
+ numbytes = (off_t)php_stream_write(stream,
Z_STRVAL_P(data),
Z_STRLEN_P(data));
+ if (numbytes != (off_t)Z_STRLEN_P(data)) {
+ php_error_docref(NULL TSRMLS_CC,
E_WARNING, Only %d of %d bytes
written, possibly out of free disk space, (size_t)numbytes,
Z_STRLEN_P(data));
numbytes = -1;
}
}
@@ -680,13 +680,13 @@
convert_to_string(*tmp);
}
if (Z_STRLEN_PP(tmp)) {
- numbytes += Z_STRLEN_PP(tmp);
+ numbytes +=
(off_t)Z_STRLEN_PP(tmp);
bytes_written =
php_stream_write(stream, Z_STRVAL_PP(tmp),
Z_STRLEN_PP(tmp));
if (bytes_written 0 ||
bytes_written != Z_STRLEN_PP(tmp)) {
if (bytes_written 0) {
php_error_docref(NULL TSRMLS_CC, E_WARNING, Failed to write
%d bytes to %s, Z_STRLEN_PP(tmp), filename);
} else {
-
php_error_docref(NULL TSRMLS_CC, E_WARNING, Only %d of %d
bytes written, possibly out of free disk space, bytes_written,
Z_STRLEN_PP(tmp));
+
php_error_docref(NULL TSRMLS_CC, E_WARNING, Only %d of %d
bytes written, possibly out of free disk space, (size_t)bytes_written,
Z_STRLEN_PP(tmp));
}
numbytes = -1;
break;
@@ -702,9 +702,9 @@
zval out;
if (zend_std_cast_object_tostring(data, out,
IS_STRING TSRMLS_CC)
== SUCCESS) {
- numbytes = php_stream_write(stream,
Z_STRVAL(out),
Z_STRLEN(out));
- if (numbytes != Z_STRLEN(out)) {
- php_error_docref(NULL
TSRMLS_CC, E_WARNING, Only %d of %d bytes
written, possibly out of free disk space, numbytes
#27792 [Com]: [PATCH] Functions fail on large files (filesize,is_file,is_dir,readdir)
ID: 27792 Comment by: mail dot pourri at laposte dot net Reported By: kode at kodekrash dot com Status: Critical Bug Type: Filesystem function related Operating System: * (LFS) PHP Version: 5.*, 6CVS (2009-04-30) New Comment: Please see fix in http://bugs.php.net/bug.php?id=48886 Previous Comments: [2009-04-30 19:45:57] j...@php.net And bug #45040 [2009-04-30 19:45:19] j...@php.net See also bug #48099 (NFS mounts misbehave also without the compile flags..) [2008-12-05 17:28:36] aklmnop at gmail dot com I had to change a couple more LVAL's to DVAL and RETURN_LONG to RETURN_DOUBLE after using the above patch, to make fseek(), fread(), ftell() and fwrite() work in PHP 5.2.4. This stuff is way too broken. Shamefully broken! [2007-10-14 22:17:19] w...@php.net You might like to try the following patch: http://news.php.net/php.internals/32767 [2005-10-24 21:16:03] hendersj at mindspring dot com I have seen problems related to this both over NFS and using a local NSS filesystem with Novell Open Enterprise Server (Linux kernel, SLES 9 is where the package orginates). The problem I have seen is more accurately described in bug 33872, which was flagged as a duplicate of this bug. Bug 33872 describes a condition where readdir() returns only the entries . and .. (in my case just . is returned) regardless of the permissions to the directory. The same code run against a reiserFS partition works just fine. I am running php 4.3.4; I have also seen it with PHP 4.2.2 when using an NFS server that has 64-bit cookie values. The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at http://bugs.php.net/27792 -- Edit this bug report at http://bugs.php.net/?id=27792edit=1
