RE: [External] Re: Zip file upload corruption on Linux
Hi Chris, > Mine is coming up on 20 years old. That's worthy of an extra slice of cake :-). > The code you posted shows imports and then your interaction with the > fileupload library. Do you know what else happens before this line of code? > ServletRequestContext requestContext = new ServletRequestContext(/* > HttpServletRequest */ request); I've now reverted my changes to the latest SVN checkin from when I adjusted the code to avoid deprecated methods to try to remedy this problem and that line no longer exists. > Can you reproduce this in a testing environment? I'll bet we can write a > Filter or Valve which can catch this bug red-handed. I'd love to have the time to do this, but my motivation to do so has all but been killed by pragmatism. I could send you the two Java classes off-list if you'd like to speculate further? Thanks, Tim - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [External] Re: Zip file upload corruption on Linux
Tim, On 5/25/21 11:22, Scott,Tim wrote: Hi Chris, "nah, nobody still uses Struts 1.x". I wouldn't put it past this 14 year old application ... :) Mine is coming up on 20 years old. But at this point, if you have things working, you can probably stop. > My OCD says No!, but my pragmatic side says "leave it until I have to change" But something is *definitely* wrong if changing the default file encoding causes your files to be corrupted. It is *extraordinarily* unlikely that Tomcat or Struts is doing this. It is much more likely to be your application somewhere writing to a Writer instead of a Stream. Whilst I haven't explored every class in detail, the classes I have been working with are the first (of my code) to receive the requests and the data I'm getting is already corrupted. For example, there's nothing in my application code which writes to a temporary file as part of this process. My code writes the data to an Oracle database, binding as a binary (RAW) value. The code you posted shows imports and then your interaction with the fileupload library. Do you know what else happens before this line of code? ServletRequestContext requestContext = new ServletRequestContext(/* HttpServletRequest */ request); If the application calls one of a series of methods on HttpServletRequest, it can cause a few things to happen: 1. A "reader" is obtained from the request, which will convert bytes -> chars. 2. The "reader" needs to know what character encoding to use. There are some rules to determine what encoding that is, but Tomcat itself will always fall-back to ISO-8859-1 (per HTTP spec) and that is the encoding which does not cause corruption for you. Can you reproduce this in a testing environment? I'll bet we can write a Filter or Valve which can catch this bug red-handed. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [External] Re: Zip file upload corruption on Linux
Hi Chris, > "nah, nobody still uses Struts 1.x". I wouldn't put it past this 14 year old application ... > But at this point, if you have things working, you can probably stop. My OCD says No!, but my pragmatic side says "leave it until I have to change" > But something is *definitely* wrong if changing the default file encoding > causes your files to be corrupted. It is *extraordinarily* unlikely that > Tomcat or Struts is doing this. It is much more likely to be your application > somewhere writing to a Writer instead of a Stream. Whilst I haven't explored every class in detail, the classes I have been working with are the first (of my code) to receive the requests and the data I'm getting is already corrupted. For example, there's nothing in my application code which writes to a temporary file as part of this process. My code writes the data to an Oracle database, binding as a binary (RAW) value. Thanks, Tim - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [External] Re: Zip file upload corruption on Linux
Tim, On 5/25/21 05:03, Scott,Tim wrote: Hi Mark, No. You should be able to use HttpServletRequest.getPart() I've given up on that attempt as I keep getting: java.lang.AbstractMethodError: Method org/apache/struts/upload/MultipartRequestWrapper.getPart(Ljava/lang/String;)Ljavax/servlet/http/Part; is abstract I have my workaround and do not anticipate it worthwhile me spending any more time on the matter. You know, it's funny. I use Struts 1.x and in order to use the Tomcat-provided multipart handling, you need to do extra work. I thought of that when Mark suggested using Tomcat's multipart parsing but then thought "nah, nobody still uses Struts 1.x". Anyway, if you want to disable Struts's multipart handling, you have to add this to your definition for Struts. Note that this may break other parts of your application that might depend upon the Struts multipart handling. But at this point, it's not working anyway, so you are probably okay. action org.apache.struts.action.ActionServlet Disable Struts Multipart Handling multipartClass none 1048576 1049600 1024 Note that you may have to "merge" the above with what you have in your WEB-INF/web.xml. But at this point, if you have things working, you can probably stop. But something is *definitely* wrong if changing the default file encoding causes your files to be corrupted. It is *extraordinarily* unlikely that Tomcat or Struts is doing this. It is much more likely to be your application somewhere writing to a Writer instead of a Stream. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [External] Re: Zip file upload corruption on Linux
Hi Mark, > No. You should be able to use HttpServletRequest.getPart() I've given up on that attempt as I keep getting: java.lang.AbstractMethodError: Method org/apache/struts/upload/MultipartRequestWrapper.getPart(Ljava/lang/String;)Ljavax/servlet/http/Part; is abstract I have my workaround and do not anticipate it worthwhile me spending any more time on the matter. Thanks, Tim - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [External] Re: Zip file upload corruption on Linux
On 24/05/2021 14:22, Scott,Tim wrote: Hi Mark, From: Mark Thomas wrote: import org.apache.commons.fileupload.disk.DiskFileItemFactory; import org.apache.commons.fileupload.servlet.ServletFileUpload; import org.apache.commons.fileupload.servlet.ServletRequestContext; You are using Commons FileUpload so this issue needs to be raised with the Apache Commons project. Alternatively, in Tomcat 9 file upload support is available via the Servlet API. You could try switching to that (and any bugs would then be a Tomcat issue). I replaced "org.apache.commons.fileupload." with "org.apache.tomcat.util.http.fileupload." and tried again. I found no change in behaviour: Leaving file.encoding to default to UTF-8 still corrupted the content. Setting it to ISO-8859-1 again resolved it. Was that the Servlet API you were meaning? No. You should be able to use HttpServletRequest.getPart() Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [External] Re: Zip file upload corruption on Linux
Hi Mark, From: Mark Thomas wrote: > import org.apache.commons.fileupload.disk.DiskFileItemFactory; > import org.apache.commons.fileupload.servlet.ServletFileUpload; > import org.apache.commons.fileupload.servlet.ServletRequestContext; > You are using Commons FileUpload so this issue needs to be raised with the Apache Commons project. > Alternatively, in Tomcat 9 file upload support is available via the Servlet API. You could try switching to that (and any bugs would then be a Tomcat issue). I replaced "org.apache.commons.fileupload." with "org.apache.tomcat.util.http.fileupload." and tried again. I found no change in behaviour: Leaving file.encoding to default to UTF-8 still corrupted the content. Setting it to ISO-8859-1 again resolved it. Was that the Servlet API you were meaning? Thanks, Tim - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [External] Re: Zip file upload corruption on Linux
On 24/05/2021 12:08, Scott,Tim wrote:
Hi Mark,
Thanks for the prompt response.
On 24/05/2021 10:58, Scott,Tim wrote:
Hi experts,
First time poster, here, so I know I'm risking not providing nearly
enough of the right information. Please let me know what I can send to
help you help me further through this.
How are you reading the uploaded file? Please provide the code that does this.
I am reading the InputStream as below:
(merged from two classes, untested, incomplete)
import org.apache.commons.fileupload.disk.DiskFileItemFactory;
import org.apache.commons.fileupload.servlet.ServletFileUpload;
import org.apache.commons.fileupload.servlet.ServletRequestContext;
You are using Commons FileUpload so this issue needs to be raised with
the Apache Commons project.
Alternatively, in Tomcat 9 file upload support is available via the
Servlet API. You could try switching to that (and any bugs would then be
a Tomcat issue).
Mark
import javax.servlet.http.HttpServletRequest;
import java.io.InputStream;
import org.apache.commons.fileupload.FileItem;
ServletRequestContext requestContext = new ServletRequestContext(/*
HttpServletRequest */ request);
FileItemFactory factory = new DiskFileItemFactory();
FileUpload fileUpload = new ServletFileUpload(factory);
List entries = fileUpload.parseRequest(requestContext); // <<< this
call generates the temp file
InputStream inputStream;
for (FileItem item : entries)
{
if (!item.isFormField())
{
inputStream = item.getInputStream();
}
}
...
byte[] buffer = new byte[BINARY_BUFFER_SIZE];
bolean eof = false;
while (!eof)
{
int count = inputStream.read(buffer);
if (count == -1)
{
eof = true;
...
}
...
}
Similarly, I am not writing the temp file. I understand that this is done by
DeferredFileOutputStream as part of the call to ServletFileUpload's
parseRequest(); The temp file (if created) and the input stream already contain
corrupted data.
The only way the default encoding should impact things is if the file bytes are
being converted to String at some point.
Not by me, they're not!
That shouldn't normally happen for an uploaded file.
I agree, it shouldn't. That does not match, however, my finding that:
Using -Dfile.encoding=utf-8 on Windows corrupts the file.
Using -Dfile.encoding=ISO-8859-1 on Linux stops the file
corruption.
Thanks,
Tim
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
RE: [External] Re: Zip file upload corruption on Linux
Hi Mark,
Thanks for the prompt response.
>On 24/05/2021 10:58, Scott,Tim wrote:
>> Hi experts,
>>
>> First time poster, here, so I know I'm risking not providing nearly
>> enough of the right information. Please let me know what I can send to
>> help you help me further through this.
>How are you reading the uploaded file? Please provide the code that does this.
I am reading the InputStream as below:
(merged from two classes, untested, incomplete)
import org.apache.commons.fileupload.disk.DiskFileItemFactory;
import org.apache.commons.fileupload.servlet.ServletFileUpload;
import org.apache.commons.fileupload.servlet.ServletRequestContext;
import javax.servlet.http.HttpServletRequest;
import java.io.InputStream;
import org.apache.commons.fileupload.FileItem;
ServletRequestContext requestContext = new ServletRequestContext(/*
HttpServletRequest */ request);
FileItemFactory factory = new DiskFileItemFactory();
FileUpload fileUpload = new ServletFileUpload(factory);
List entries = fileUpload.parseRequest(requestContext); // <<< this
call generates the temp file
InputStream inputStream;
for (FileItem item : entries)
{
if (!item.isFormField())
{
inputStream = item.getInputStream();
}
}
...
byte[] buffer = new byte[BINARY_BUFFER_SIZE];
bolean eof = false;
while (!eof)
{
int count = inputStream.read(buffer);
if (count == -1)
{
eof = true;
...
}
...
}
Similarly, I am not writing the temp file. I understand that this is done by
DeferredFileOutputStream as part of the call to ServletFileUpload's
parseRequest(); The temp file (if created) and the input stream already contain
corrupted data.
> The only way the default encoding should impact things is if the file bytes
> are being converted to String at some point.
Not by me, they're not!
> That shouldn't normally happen for an uploaded file.
I agree, it shouldn't. That does not match, however, my finding that:
Using -Dfile.encoding=utf-8 on Windows corrupts the file.
Using -Dfile.encoding=ISO-8859-1 on Linux stops the file
corruption.
Thanks,
Tim
