We are currently working on a scheme to successfully authenticate and verify the integrity of the data. Datasets in https://climate.daknob.net/ are compressed to a .tar.bz2 and then hashed using SHA-256. The final file with all checksums is then signed using a set of PGP keys.
We are still working on a viable way to verify the authenticity of files before there are tons of copies lying around and there’s a working group in the Slack team I sent previously where your input is much needed! Thanks, Antonios > On 16 Dec 2016, at 18:30, Ken Chase <[email protected]> wrote: > > Surfing through the links - any hints on how big these datasets are? > Everyone's got > a few TB to throw at things, but fewer of us have spare PB to throw around. > > There's some random #s on the goog doc sheet for sizes (100's of TB for the > landsat archive seems credible), and there's one number that destroys > credibility of the sheet (100000000000 GB (100 ZB)) for the EPA archive. > > The other page has many 'TBA' entries for size. > > Not sure what level of player one needs to be to be able to serve a useful > segment of these archives. I realize some of the datasets are tiny (<GB) > but which ones are most important vs size (ie the win-per-byte ratio) isnt > indicated. > (I know its early times.) > > Also I hope they've SHA512'd the datasets for authenticity before all these > myriad copies being flungabout are 'accused' of being manipulated 'to promote > the climate change agenda' yadda. > > Canada: time to step up! (Cant imagine the Natl Research Council would do so > on their mirror site, too much of a gloves-off slap in the face to Trump.) > > /kc > > > On Fri, Dec 16, 2016 at 06:02:46PM +0200, DaKnOb said: >> If you???re interested, there???s also a Slack team: climatemirror.slack.com >> >> You can find more info about that here: >> >> - https://climate.daknob.net/ >> - http://climatemirror.org/ >> - http://www.ppehlab.org/datarefuge >> >> Thank you for your help! >> >> >>> On 16 Dec 2016, at 17:58, Rich Kulawiec <[email protected]> wrote: >>> >>> This is a short-term (about one month) project being thrown together >>> in a hurry...and it could use some help. I know that some of >>> you have lots of resources to throw at this, so if you have an >>> interest in preserving a lot of scientific research data, I've set >>> up a mailing list to coordinate IT efforts to help out. Signup via >>> [email protected] or, if you prefer Mailman's web >>> interface, http://www.firemountain.net/mailman/listinfo/climatedata >>> should work. >>> >>> Thanks, >>> ---rsk >>> >> > > -- > Ken Chase - [email protected] Guelph Canada
