Re: [galaxy-user] Problems with large gzipped fasta files
Hi Jim, Could you send me a URL to the dataset so I can grab a copy and try to reproduce this problem? Sorry for the trouble you've been having with the upload functionality and the delay in getting back to you. --nate On Feb 5, 2013, at 8:48 AM, Jim Robinson wrote: Hi, I am having a lot of difficulty uploading some large gzipped fastqs (~ 10GB) to the public server. I have tried both ftp and pulling by http URL. The upload succeeds, however I get an error as it tries to gunzip it.I have tried more than 10 times now and succeeded once. These files are correct and complete, and gunzip properly locally. The error shown is usually this empty format: txt, database: ? Problem decompressing gzipped data However on 2 occasions (both ftp uploads) I got the traceback below. Am I missing some obvious trick? I searched the archives and see references to problems with large gzipped files but no solutions. Thanks Jim Traceback (most recent call last): File /galaxy/home/g2main/galaxy_main/tools/data_source/upload.py, line 384, in module __main__() File /galaxy/home/g2main/galaxy_main/tools/data_source/upload.py, line 373, in __main__ add_file( dataset, registry, json_file, output_path ) File /galaxy/home/g2main/galaxy_main/tools/data_source/upload.py, line 270, in add_file line_count, converted_path = sniff.convert_newlines( dataset.path, in_place=in_place ) File /galaxy/home/g2main/galaxy_main/lib/galaxy/datatypes/sniff.py, line 106, in convert_newlines shutil.move( temp_name, fname ) File /usr/lib/python2.7/shutil.py, line 299, in move copy2(src, real_dst) File /usr/lib/python2.7/shutil.py, line 128, in copy2 copyfile(src, dst) File /usr/lib/python2.7/shutil.py, line 84, in copyfile copyfileobj(fsrc, fdst) File /usr/lib/python2.7/shutil.py, line 49, in copyfileobj buf = fsrc.read(length) IOError: [Errno 5] Input/output error ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-user] Problems with large gzipped fasta files
Sorry Nate, I misunderstood at first, you want a URL to the dataset here on my server? I can definitely copy one up to an http server, I still have Ricardo's files on a hard disk. I'll start the copy now and let you know when its ready. Jim Hi Jim, Could you send me a URL to the dataset so I can grab a copy and try to reproduce this problem? Sorry for the trouble you've been having with the upload functionality and the delay in getting back to you. --nate On Feb 5, 2013, at 8:48 AM, Jim Robinson wrote: ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-user] Problems with large gzipped fasta files
On Feb 13, 2013, at 12:25 PM, Jim Robinson wrote: Sorry Nate, I misunderstood at first, you want a URL to the dataset here on my server? I can definitely copy one up to an http server, I still have Ricardo's files on a hard disk. I'll start the copy now and let you know when its ready. Yeah, that's it exactly. Thanks! --nate Jim Hi Jim, Could you send me a URL to the dataset so I can grab a copy and try to reproduce this problem? Sorry for the trouble you've been having with the upload functionality and the delay in getting back to you. --nate On Feb 5, 2013, at 8:48 AM, Jim Robinson wrote: ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
Re: [galaxy-user] Problems with large gzipped fasta files
Hi Jim, You message was misthreaded (perhaps a reply to another thread, with just the subject line changed?), but I was able to dig it out. A this time, there are no known issues with FTP Upload to the public Main server. Any issues you have have found prior were either related to a problem with the original file content (compression problem) or a transitory issue with the FTP server that has since been resolved (there has been a handful in the last few years). The instructions to follow are here: http://wiki.galaxyproject.org/FTPUpload I am not exactly sure what your issue is, but any chance that you have more than one file per archive? That will certainly cause an issue, but usually with just the first file loading the remainder not. Please send more details if this continues. Does the failure occur at the FTP stage or at the point where you move from the FTP holding area into a history? Thanks! Jen Galaxy team On 2/5/13 5:48 AM, Jim Robinson wrote: Hi, I am having a lot of difficulty uploading some large gzipped fastqs (~ 10GB) to the public server. I have tried both ftp and pulling by http URL. The upload succeeds, however I get an error as it tries to gunzip it.I have tried more than 10 times now and succeeded once. These files are correct and complete, and gunzip properly locally. The error shown is usually this empty format: txt, database: ? Problem decompressing gzipped data However on 2 occasions (both ftp uploads) I got the traceback below. Am I missing some obvious trick? I searched the archives and see references to problems with large gzipped files but no solutions. Thanks Jim Traceback (most recent call last): File /galaxy/home/g2main/galaxy_main/tools/data_source/upload.py, line 384, in module __main__() File /galaxy/home/g2main/galaxy_main/tools/data_source/upload.py, line 373, in __main__ add_file( dataset, registry, json_file, output_path ) File /galaxy/home/g2main/galaxy_main/tools/data_source/upload.py, line 270, in add_file line_count, converted_path = sniff.convert_newlines( dataset.path, in_place=in_place ) File /galaxy/home/g2main/galaxy_main/lib/galaxy/datatypes/sniff.py, line 106, in convert_newlines shutil.move( temp_name, fname ) File /usr/lib/python2.7/shutil.py, line 299, in move copy2(src, real_dst) File /usr/lib/python2.7/shutil.py, line 128, in copy2 copyfile(src, dst) File /usr/lib/python2.7/shutil.py, line 84, in copyfile copyfileobj(fsrc, fdst) File /usr/lib/python2.7/shutil.py, line 49, in copyfileobj buf = fsrc.read(length) IOError: [Errno 5] Input/output error ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ -- Jennifer Hillman-Jackson Galaxy Support and Training http://galaxyproject.org ___ The Galaxy User list should be used for the discussion of Galaxy analysis and other features on the public server at usegalaxy.org. Please keep all replies on the list by using reply all in your mail client. For discussion of local Galaxy instances and the Galaxy source code, please use the Galaxy Development list: http://lists.bx.psu.edu/listinfo/galaxy-dev To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/
