Re: [galaxy-dev] Careful data migration
Hi Sebastian I am also not aware of any such documentation. It is probably impossible to write it, as each such situation is different. I don't quite understand what your 'script' is doing, but in your situation, I recommend the following: First of all, make sure you have a backup of your PostgreSQL (or MySQL) database. Don't make any changes to 'universe.wsgi', but move the bulky directories to new locations on the NFS share and replace them with symbolic links. This has also the advantage, that you can do it step by step (ie directory by directory). Regards, Hans-Rudolf On 01/13/2014 05:33 PM, Sebastian Schaaf wrote: Hi @ all, I would like to know if anyone could give me some guidance or hint on how to migrate data 'correctly' if Galaxy already wrote files, worked on the database etc. The situation is that we set up an instance, it was already in use and wrote files e.g. into the directory 'galaxy-dist/database/'. The local space is quite small, but more storage is available via a NFS share. I already wrote a bash script for setting up the connections (and editing the 'universe.wsgi' file accordingly) for the entries 'genome_data_path, 'ftp_upload_dir', 'file_path', 'new_file_path' and 'job_working_directory'. Those should address the most bulky features... The application of that script is fine as long as the Galaxy system has never started before. While applying that on a system that already 'did something', e.g. the files in the 'database/' subdirectory remain there and are not transferred. Anyhow, if Galaxy is started up after the application of the new settings, no error is reported. I wonder how Galaxy now deals with that situation: * Handling data from both sources? (read and/or write?) * Automatic movements of the 'historic' data when touched the next time? * Crashes if those older objects are intended to be read/edited? * Removes them silently from the database? = Is there a procedure/module to used in order to migrate the data? Is it sufficient (and appropriate) just to move all contents of the old folders to the new locations? Did I miss any existing documentation on that issue? The answer(s) may diverge, depending on to which of the five parameters announced above those questions are addressed... Any help appreciated before blowing up our production instance :). Thanks in advance, Best regards, Sebastian ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
Re: [galaxy-dev] Careful data migration
Hi Hans-Rudolph, Thanks for that. I think you are right that there is no peace of code for that, but it could have been the case (no intend to trigger something like 'rtm' ;) ). My script basically creates the NFS mount points, mounts the network target directories, shuts down Galaxy if running, edits the universe.wsgi file according to the new directories (here: the mount points, according to those five 'bulky parameters') and starts Galaxy again if previously running. The primary purpose is to setup a completely new system from scratch (the NFS section is just a part of a longer script), but I would like to redirect the bulky directories also in our main instance (which is in use). The symbolic links are an adequate workaround I was already aware of, but anyway I am interested in what other developers may have experienced. The situation should not be too exotic..? Best, Sebastian Hans-Rudolf Hotz schrieb: Hi Sebastian I am also not aware of any such documentation. It is probably impossible to write it, as each such situation is different. I don't quite understand what your 'script' is doing, but in your situation, I recommend the following: First of all, make sure you have a backup of your PostgreSQL (or MySQL) database. Don't make any changes to 'universe.wsgi', but move the bulky directories to new locations on the NFS share and replace them with symbolic links. This has also the advantage, that you can do it step by step (ie directory by directory). Regards, Hans-Rudolf On 01/13/2014 05:33 PM, Sebastian Schaaf wrote: Hi @ all, I would like to know if anyone could give me some guidance or hint on how to migrate data 'correctly' if Galaxy already wrote files, worked on the database etc. The situation is that we set up an instance, it was already in use and wrote files e.g. into the directory 'galaxy-dist/database/'. The local space is quite small, but more storage is available via a NFS share. I already wrote a bash script for setting up the connections (and editing the 'universe.wsgi' file accordingly) for the entries 'genome_data_path, 'ftp_upload_dir', 'file_path', 'new_file_path' and 'job_working_directory'. Those should address the most bulky features... The application of that script is fine as long as the Galaxy system has never started before. While applying that on a system that already 'did something', e.g. the files in the 'database/' subdirectory remain there and are not transferred. Anyhow, if Galaxy is started up after the application of the new settings, no error is reported. I wonder how Galaxy now deals with that situation: * Handling data from both sources? (read and/or write?) * Automatic movements of the 'historic' data when touched the next time? * Crashes if those older objects are intended to be read/edited? * Removes them silently from the database? = Is there a procedure/module to used in order to migrate the data? Is it sufficient (and appropriate) just to move all contents of the old folders to the new locations? Did I miss any existing documentation on that issue? The answer(s) may diverge, depending on to which of the five parameters announced above those questions are addressed... Any help appreciated before blowing up our production instance :). Thanks in advance, Best regards, Sebastian -- Sebastian Schaaf, M.Sc. Bioinformatics Faculty Coordinator NGS Infrastructure Chair of Biometry and Bioinformatics Department of Medical Informatics, Biometry and Epidemiology (IBE) University of Munich Marchioninistr. 15, K U1 (postal) Marchioninistr. 17, U 006 (office) D-81377 Munich (Germany) Tel: +49 89 2180-78178 ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
[galaxy-dev] Careful data migration
Hi @ all, I would like to know if anyone could give me some guidance or hint on how to migrate data 'correctly' if Galaxy already wrote files, worked on the database etc. The situation is that we set up an instance, it was already in use and wrote files e.g. into the directory 'galaxy-dist/database/'. The local space is quite small, but more storage is available via a NFS share. I already wrote a bash script for setting up the connections (and editing the 'universe.wsgi' file accordingly) for the entries 'genome_data_path, 'ftp_upload_dir', 'file_path', 'new_file_path' and 'job_working_directory'. Those should address the most bulky features... The application of that script is fine as long as the Galaxy system has never started before. While applying that on a system that already 'did something', e.g. the files in the 'database/' subdirectory remain there and are not transferred. Anyhow, if Galaxy is started up after the application of the new settings, no error is reported. I wonder how Galaxy now deals with that situation: * Handling data from both sources? (read and/or write?) * Automatic movements of the 'historic' data when touched the next time? * Crashes if those older objects are intended to be read/edited? * Removes them silently from the database? = Is there a procedure/module to used in order to migrate the data? Is it sufficient (and appropriate) just to move all contents of the old folders to the new locations? Did I miss any existing documentation on that issue? The answer(s) may diverge, depending on to which of the five parameters announced above those questions are addressed... Any help appreciated before blowing up our production instance :). Thanks in advance, Best regards, Sebastian -- Sebastian Schaaf, M.Sc. Bioinformatics Faculty Coordinator NGS Infrastructure Chair of Biometry and Bioinformatics Department of Medical Informatics, Biometry and Epidemiology (IBE) University of Munich ___ Please keep all replies on the list by using reply all in your mail client. To manage your subscriptions to this and other Galaxy lists, please use the interface at: http://lists.bx.psu.edu/ To search Galaxy mailing lists use the unified search at: http://galaxyproject.org/search/mailinglists/
