As a note on practice, NASA's ERBE and CERES projects (the latter continuing on JPSS) use a hierarchical categorization of files. CERES, in particular, is likely to continue indefinitely.
Note also that sites archiving rock core samples fit into this category, as does the NOAA Emergency Responder Imagery Collection (ERIC). ERIC only picks up one or two storms per year and they already handle very short turnaround on 100,000 images taken a day or two after a major storm. Bruce B. On Thu, Apr 23, 2015 at 1:11 AM, Mattmann, Chris A (3980) < [email protected]> wrote: > Hey ThomasB (we have 2 “TomB”s now, so I’ll fully use your > name ;) ) > > Great questions. My thoughts below: > > > > -----Original Message----- > From: Thomas Bennett <[email protected]> > Reply-To: "[email protected]" <[email protected]> > Date: Wednesday, April 22, 2015 at 6:09 AM > To: OODT <[email protected]> > Subject: Datastore references for a duplicate products > > >Hi, > > > >Okay - so a 'flat' product has a single datastore reference. > > > >So, how do you handle redundant copies of products? At the moment I have > >an > >independent catalogue at each site. > > > >I was thinking of a site metadata key, so multiple products can be > >filtered, but I thought I would see what other people are doing and if > >there is any interest in perhaps getting a product (flat or heirachical) > >with multiple references, i.e. beyond originalReference and > >datastoreReference. > > My personal opinion is that tagging each Product with a “Site” metadata > field would be a great idea and a great way to filter this later. > > > > >Or does that totally break the OODT model? It probably does... > > Nope doesn’t break the model at all - the Product with structure > ProductStructure.HIERARCHICAL works fine with this - data transfers > work fine - metadata retrieval works fine (try ingesting a directory), > and other things work fine too. > > However, I would seriously advise *against* using this model ^_^ > Mostly b/c no one really does it that way - and the code isn’t actively > being developed and maintained. At one point I thought it would be a > great thing to support and the last I heard of someone using this structure > natively was OCO from 2007/2008 (a NASA mission). > > > > >Also - does anyone store data on tape library and index it with OODT. I'm > >talking basic tar on a tape. This obviously breaks the file retrieval, if > >used, but I'm thinking of how this could be included in the OODT framework > >and maybe develop some methods. > > Well yep we do this - one way to do would be to add a > TapeBasedDataTransferer, > and/or utilities to handle waiting for data e.g., from Tape. You could also > use the InPlaceDataTransferer, and then come up with a Versioner that > easily > supports having Tape-based URIs. > > Does that make sense? > > Cheers, > Chris > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Chris Mattmann, Ph.D. > Chief Architect > Instrument Software and Science Data Systems Section (398) > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA > Office: 168-519, Mailstop: 168-527 > Email: [email protected] > WWW: http://sunset.usc.edu/~mattmann/ > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Adjunct Associate Professor, Computer Science Department > University of Southern California, Los Angeles, CA 90089 USA > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > >
