Hi Benjamin, I don't have a quick fix, I don't know if others have seen this kind of issue before. I've created a ticket in Trac for it: https://www.orangefs.org/trac/orangefs/ticket/34
I'm planning on just copying your setup to recreate the problem. Anything special about the contents of the 10GB file or just urandom type data? Thanks for letting us know about this! Michael On Tue, Jun 7, 2011 at 9:37 AM, Benjamin Severs <[email protected]>wrote: > When FTPing a file to an FTP server that exposed an OrangeFS file system, I > can consistently get data corruption when the ProFTPD server has a specific > configuration. If I set the send and > recv socket buffer size settings for ProFTPD (as configured in the attached > config file), it will often corrupt the data. As far as I could determine, > all these settings do is modify the socket > options for the FTP sockets using SO_SNDBUF and SO_RCVBUF options set with > setsockopt. > > Here is my environment configuration: > > FTP server: ProFTPD (I just built and installed the latest version > available at proftpd.org). I've attached the configuration file. > FTP client: The standard "ftp" command line utility present in RedHat Linux > FS Cluster: 4 Node OrangeFS Cluster using the attached config file. > All nodes involved were running 64bit RHEL5.5 > > Here is the command line used for the pvfs2-client: > /usr/sbin/pvfs2-client --logtype syslog -p /usr/sbin/pvfs2-client-core > --logstamp datetime --acache-timeout=30000 --ncache-timeout=30000 > --desc-size 8388608 --desc-count 5 > > The FTP client and server were on the same machine, so I simply FTP'd to > localhost. > > We initially saw the corruption when copying a 110G file, but a 118G file > copied just fine. I've been able to reliably reproduce the corruption with > a 10G file and sometimes with a 1G file. I was > never able to reproduce the corruption with a file smaller than 500M. > > The corruption itself seems to manifest itself by replacing good data with > Nulls. The file size is always correct. > > Another observation is that it usually takes several attempts for a file > transfer to cause corruption, but once it does, it is fairly consistent. > > Also, I attempted to reproduce the issue by using the "cp" command, "dd", > and "rsync". None of these operations would reproduce the issue. Even > using the "curl" command line utility to > make the FTP transfer worked. The only cases that cause corruption are our > custom FTP client and the command line ftp client. I'm assuming their > access pattern triggers some edge case in > OrangeFS. > > I also put debug statements in the proftpd code to write the data to a > local file immediately before writing to the OrangeFS FS. The debug file > would be correct (without corruption), while the > file written out to OrangeFS would be corrupted, so this isn't an issue of > the data being corrupted in-flight to the FTP server. > > Any help you can provide would be greatly appreciated. > > Thanks! > > -- > Benjamin Severs > > _______________________________________________ > Pvfs2-developers mailing list > [email protected] > http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers > >
_______________________________________________ Pvfs2-developers mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
