Apache::Upload bug?

[Jeffrey Hartmann]

Ok, for a the last couple days I've been searching out a problem I've been
having with Apache::Upload and Image::Magick.

The code consists mainly of::

@upload = $r-upload;
 foreach my $file (@upload)
{
my $fh = $file-fh;

# There's some .ext checking here to get $type and some renaming.

my $i = Image::Magick-new(magick=$type);
$err = $i-ReadImage(file=$$fh);
$i-Set(magick=$type);
$err = $i-WriteImage(filename=$filename));
warn $err if $err;
undef $i;
}


Now the main problem was that when the request was over the temp file
Apache::Upload creates, apreq??, in the /tmp directory was getting
unlink'd but there was still an open filehandle to it.  This meant that the
space the image was taking up on the /tmp dir was not being cleared, but the
file wasn't showing up in the directory.  And each image upload after this
was creating more stale filehandles and keeping more and more drive space
occupied.  If Apache was restarted the filehandles were closed and the
memory was free'd.  I assume when a child dies it also clears the memory,
but I'm not sure on that one.

After much tracing I found that the problem occurs in the command my $fh =
$file-fh; where Apache::Upload dup()'s the filehandle and passed the
duplicate to the perl script.  Then when the program exits the perl script
still has an open filehandle to that file.  I fixed my problem by adding a
close $$fh;  to my program which closed the duplicate filehandle.

Now I'm not sure if this is a bug or if it's supposed to be like that, but
the documentation makes it sound like it gives you the actually filehandle
of the tempfile, and not a copy.  I just assumed that it would be closed by
Apache::Upload when the request was finished.



-Jeff Hartmann

Re: Apache::Upload and Image::Magick problems

[Jeffrey Hartmann]

Thanks Joe!!

Well that was exactly right.  The Apache::Upload temp file is still open (or
mmap'd) after it's unlinked and ImageMagick just doesn't delete it's file.
I don't have the problem with Image Magick with small files because it
doesn't create temporary files for small images.  I also checked the trace
and ImageMagick makes no attempt to delete it's temporary file.  In fact it
seems to use 2 temporary files (which means I need filesize * 2 space to
upload an image) and it deletes one and not the other.  Apache::Upload
unlinks the the file and un-mmaps it but it just doesn't seem to do the
trick.

I've tried using this with and without the TMPFS and I get the same result
either way.

However now I really don't know what to do.  I was hoping that it was some
bonehead error in my code that could easily be fixed.  Maybe it is... but it
doesn't seem like it now.


-Jeff Hartmann


Note: I have it upload five files at a time.
- lsof ---

COMMANDPID USER   FD   TYPE DEVICESIZE  NODE NAME
httpd 1660 root5u   REG0,6 9449700 28525
/tmp/apreqT2TdFO (deleted)
httpd 1660 root6u   REG0,6   0 28526
/tmp/apreqDUBnju (deleted)
httpd 1660 root7u   REG0,6   0 28527
/tmp/apreqB3QPJb (deleted)
httpd 1660 root8u   REG0,6   0 28528
/tmp/apreqf8YOdT (deleted)
httpd 1660 root9u   REG0,6   0 28529
/tmp/apreqpXsJPC (deleted)


--- strace httpd -X -

close(36)   = 0
munmap(0x403a9000, 4096)= 0
rmdir(/tmp/apreqpXsJPC)   = -1 ENOTDIR (Not a directory)
unlink(/tmp/apreqpXsJPC)  = 0
close(36)   = -1 EBADF (Bad file descriptor)
close(35)   = 0
munmap(0x403a2000, 4096)= 0
rmdir(/tmp/apreqf8YOdT)   = -1 ENOTDIR (Not a directory)
unlink(/tmp/apreqf8YOdT)  = 0
close(35)   = -1 EBADF (Bad file descriptor)
close(34)   = 0
munmap(0x403a1000, 4096)= 0
rmdir(/tmp/apreqB3QPJb)   = -1 ENOTDIR (Not a directory)
unlink(/tmp/apreqB3QPJb)  = 0
close(34)   = -1 EBADF (Bad file descriptor)
close(33)   = 0
munmap(0x403a, 4096)= 0
rmdir(/tmp/apreqDUBnju)   = -1 ENOTDIR (Not a directory)
unlink(/tmp/apreqDUBnju)  = 0
close(33)   = -1 EBADF (Bad file descriptor)
close(32)   = 0
munmap(0x4039f000, 4096)= 0
rmdir(/tmp/apreqT2TdFO)   = -1 ENOTDIR (Not a directory)
unlink(/tmp/apreqT2TdFO)  = 0
close(32)   = -1 EBADF (Bad file descriptor)

- END OF PROGRAM OUTPUT ---


- Original Message -
From: Joe Schaefer [EMAIL PROTECTED]
To: Jeffrey Hartmann [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Sent: Monday, August 06, 2001 7:38 AM
Subject: Re: Apache::Upload and Image::Magick problems


 Jeffrey Hartmann [EMAIL PROTECTED] writes:

  2).  Apache::Upload seams to delete it's temp file, however when I run
df
  the memory that file used is still allocated but there are no files in
the
  /tmp dir.  I've commented out all of the Image::Magick code in that
block so
  that Image::Magick never uses or accesses the file and the allocated
memory
  problem still exists.  So this means that I end up with 100 meg of used
  space with only half that showing up as files in the dir.  I can delete
all
  the Image::Magick files in the dir, but that's only half of the space
used.
 
  Now I used to use a normal /tmp dir for this, and I still had the
  Image::Magick problem, but I never noticed the Apache::Upload one until
now.
  So I don't know if this problem exists without with tmpfs.  Also if I
  restart/graceful Apache it seems to release the disk space that it's
  holding.

 It sounds like the apreq? file in /tmp is remove()d at the end of the
 request, but the file somehow stays open until the process exits. Using
 lsof or running a stack trace on httpd -X should tell you if this is
 the case.  The relevant cleanup code is the remove_tmpfile() function in
 c/apache_request.c. If the ap_pfclose() call is failing, it should show
 up in your apache logs.

 If that's not the problem, maybe there's a persistent Apache::Upload
 object somewhere in your code?
 [...]

   foreach my $file (@upload)
  {
 
  my $i = Image::Magick-new(magick=$type);
  $err = $i-ReadImage(file=$$fh);
  $i-Set(magick=$type);
  $err = $i-WriteImage(filename=$filename));
  warn $err if $err;
  undef $i;
  }
 

 Try appending

   undef @upload;

 to make sure there are no Apache::Upload objects still floating
 around when your handler exits.

 HTH
 --
 Joe