Re: mod_python.util.StorageField.read_to_boundary has problems in 3.1 and 3.2

Sun, 06 Nov 2005 13:54:44 -0800

Nicolas,

Not that I'm the one to give permission whether to integrate things or not, but just to let you know I don't even have svn installed so I won't do it. At least not for a while...

BTW, if there are some cherrypy developers in this mailing list, the CherryPy function that handles file uploads DOES also has the same issue. 

I'm not subscribe to CherryPy dev group thus the cross posting.


    24      def read_lines_to_outerboundary(self):
    25          """Internal: read lines until outerboundary."""
    26          next = "--" + self.outerboundary
    27          last = next + "--"
    28          delim = ""
    29          last_line_lfend = True
    30          while 1:
    31              line = self.fp.readline(1<<16)
    32              if not line:
    33                  self.done = -1
    34                  break
    35              if line[:2] == "--" and last_line_lfend:
    36                  strippedline = line.strip()
    37                  if strippedline == next:
    38                      break
    39                  if strippedline == last:
    40                      self.done = 1
    41                      break
    42              odelim = delim
    43              if line[-2:] == "\r\n":
    44                  delim = "\r\n"
    45                  line = line[:-2]
    46                  last_line_lfend = True
    47              elif line[-1] == "\n":
    48                  delim = "\n"
    49                  line = line[:-1]
    50                  last_line_lfend = True
    51              else:
    52                  delim = ""
    53                  last_line_lfend = False
    54              self.__write(odelim + line)

/amn

On Nov 6, 2005, at 4:20 PM, Nicolas Lehuen wrote:

OK, it looks like Alexis' fix solves the problem with ugh.pdf without breaking the other unit tests. So I think we can safely integrate his patch. Shall I do it ?

Regards,
Nicolas

2005/11/6, Nicolas Lehuen <[EMAIL PROTECTED]>:
Hi guys,

In the pure "if it ain't tested, it ain't fixed" fashion, I've added a unit test for file upload to the test suite. It uploads a randomly generated 1 MB file to the server, and check that the MD5 digest returned by the server is correct. I could not reproduce Alexis' bug report this way, however. But I then I added a test with the UNIX-HATERS handbook file ugh.pdf, and bang, here comes the bug.

I've checked in both unit tests into subversion, so that you can play with them. I'm now going to test Alexis' fix.

Regards,
Nicolas

2005/11/6, Alexis Marrero <[EMAIL PROTECTED]>:
I don't have a function that creates the files but the I can point
you to a file that has the problem, ironically is "Unix Haters
Handbook" :) Well, at least is not the Python HH....

http://research.microsoft.com/~daniel/uhh-download.html

It's MD5 is 9e8c42be55aac825e7a34d448044d0fe. I don't know what it
ends up been after upload with read_to_boundary().

When you use the function to copy the file you will see that the
digest will be e45979254297b0ece9c182a789d7966e.

I have other 5 files out of 78546 files that I'm testing it against
that have the same issues, coincidentally there are all PDF files.
Here is the script that I was testing it with.

def read_to_boundary(self, req, boundary, file):
     ''' read from the request object line by line with a maximum size,
         until the new line starts with boundary
     '''
     previous_delimiter = ''
     while 1:
         line = req.readline(1<<16)
         if line.startswith(boundary):
             break

         if line.endswith('\r\n'):
             file.write(previous_delimiter + line[:-2])
             previous_delimiter = '\r\n'

         elif line.endswith('\r') or line.endswith('\n'):
             file.write(previous_delimiter + line[:-1])
             previous_delimiter = line[-1:]

         else:
             file.write(previous_delimiter + line)
             previous_delimiter = ''

#f = file('Perl Bookshelf [4th Ed]/mre/final/ch06.pdf.new', 'a+')
#f = file('Pages User Guide.app/Contents/Resources/Italian.lproj/
Pages Manuale Utente.pdf', 'a+')
f = file('ugh.pdf.new', 'a+')
f.write('\r\n--myboundary--\r\n')
f.seek(0)
o = file('test.bin', 'wb')
read_to_boundary(None, f, '--myboundary', o)
o.close()

/amn


On Nov 6, 2005, at 11:58 AM, Jim Gallacher wrote:

> Alexis,
>
> I wanted to add that I'm testing your code.
>
> Alexis Marrero wrote:
>
>> Let me know any comments on it and if you test it and fails please
>> also let me know. I don't have subversion account neither I don't
>> know how to use it thus this email.
>>
>
> You don't need an account to use subversion anonymously. Just
> install subversion and grab a mod_python working copy.
>
> $ svn co http://svn.apache.org/repos/asf/httpd/mod_python/trunk trunk
>
> This will checkout a working copy into a new directory called
> "trunk" on  your machine. All of the following commands assume you
> are working in trunk/.
>
> Make your changes in your working copy, and then create a diff with:
>
> $ svn diff lib/python/mod_python/util.py > your-patch.diff
>
> The other commands which you'll find immediately useful are:
>
> svn update    - update your working copy from the repository
> svn status    - shows status of changes in your working copy
> svn -u status    - shows status of your copy against the repository
>
> I've found "Version Control with Subverion" is an excellent
> resource and is available online.
> http://svnbook.red-bean.com/en/1.1/index.html
>
> Jim
>




Reply via email to