>> Sure and there are very many similar things. But still, these are >> applicable on a distributed environment (HPC, the cloud, whatever), >> not your laptop, which is what I was talking about. > > If you can't fit the whole dataset on your laptop, I don't think > there's any getting around that ;). IPFS (and likely a number of > other tools) just make it easy to pull the bits you need from the > larger dataset to your local machine (so you can run your local > analysis), while still providing a way to securely identify both the > dataset as a whole and the subset(s) you accessed.
I don't think the Merkle hashes approach would work with the amount of data we are talking about in the atmospheric science realm (namely thousands of terabytes) and in the way it's usually used. But as you said, the "data provider" can offer some subsetting as well as some minimal functionalities, e.g. finding or counting the maxima. _______________________________________________ Discuss mailing list [email protected] http://lists.software-carpentry.org/mailman/listinfo/discuss_lists.software-carpentry.org
