I don’t have an answer, but I’ll be really interested to hear your experiences. I’m in a slightly similar situation, in that I use git to store the inputs to hydrodynamic models – but while that worked nicely when I only had small inputs, some of the inputs are now many gigabytes and that ain’t gonna work nicely without… one of the things that you mention!
So if you do investigate, please report back to the list! From: thompson.m.j via discuss <[email protected]> Sent: 20 July 2018 17:08 To: discuss <[email protected]> Subject: [discuss] Version control and collaboration with large datasets. Hello all, I am a member of a computational biology lab that models processes in developmental biology and cell signaling and calibrates these models with microscopy data. I've recently gotten into using version control using git for our codes, and I am now trying to determine the best course of action to take for the data. These are the tools I'm aware of but have not tested: The Dat Project https://datproject.org/ Git Large File Storage https://git-lfs.github.com/ Git Annex https://git-annex.branchable.com/ Data Version Control (DVC) https://dvc.org/ All projects seem to be aimed at researchers trying to integrate data versioning into their workflow and collaboration, and some seem to have a few other bells and whistles. Now, the only reason I settled on using git for my work is that it seems to be the de facto standard version control just about the whole world uses. Using this same reasoning, does anyone here have a keen insight into which of the data versioning tools listed here or otherwise is (or will most likely become) the standard for data version control? The Carpentries<https://carpentries.topicbox.com/latest> / discuss / see discussions<https://carpentries.topicbox.com/groups/discuss> + participants<https://carpentries.topicbox.com/groups/discuss/members> + delivery options<https://carpentries.topicbox.com/groups/discuss/subscription> Permalink<https://carpentries.topicbox.com/groups/discuss/Tb776978a905c0bf8-M26854e6b9b3500ea27de1bc9> ________________________________ Heriot-Watt University is The Times & The Sunday Times International University of the Year 2018 Founded in 1821, Heriot-Watt is a leader in ideas and solutions. With campuses and students across the entire globe we span the world, delivering innovation and educational excellence in business, engineering, design and the physical, social and life sciences. This email is generated from the Heriot-Watt University Group, which includes: 1. Heriot-Watt University, a Scottish charity registered under number SC000278 2. Edinburgh Business School a Charity Registered in Scotland, SC026900. Edinburgh Business School is a company limited by guarantee, registered in Scotland with registered number SC173556 and registered office at Heriot-Watt University Finance Office, Riccarton, Currie, Midlothian, EH14 4AS 3. Heriot- Watt Services Limited (Oriam), Scotland's national performance centre for sport. Heriot-Watt Services Limited is a private limited company registered is Scotland with registered number SC271030 and registered office at Research & Enterprise Services Heriot-Watt University, Riccarton, Edinburgh, EH14 4AS. The contents (including any attachments) are confidential. If you are not the intended recipient of this e-mail, any disclosure, copying, distribution or use of its contents is strictly prohibited, and you should please notify the sender immediately and then delete it (including any attachments) from your system. ------------------------------------------ The Carpentries: discuss Permalink: https://carpentries.topicbox.com/groups/discuss/Tb776978a905c0bf8-Mdef0f116ac8073d883b70fd3 Delivery options: https://carpentries.topicbox.com/groups/discuss/subscription
