Hi Albert, Thanks. Sorry it took me so long to get back to you on this, replies inline below:
> > Perhaps it would be useful to summarize/rewind to the beginning. > > Summary: > > The underlying discussion is whether we want to aggregate incoming data > files both in time (32 sec -> 8 min) and across data types (something like > 10 -> 1) in order to reduce the number of mission data files we need to > handle in our system. > > Specifically the proposal is that we need to do the above 150-1 file > aggregation WITHOUT ever ingesting the 150 input files into our file > catalog. To achieve this, why not maintain a separate FM instance (call it "shadow"), configured as follows: 1. NoOp FM catalog implementation - in essence, implement the o.a.oodt.cas.filemgr.catalog.Catalog interface, but make it do nothing, i.e., no storage of prod references or metadata, no querying, nada. 2. Data Transfer and Versioning work fine, and place the 150 files you want in your desire archive location. And then have a crawler PreConditionComparator that ignores all files but the 1 file you care about, and then an associated crawler action that separately ingests into your "shadow" FM (i.e. move but not catalog). > > This is because there is concern about overloading the PCS. > > This is your "jumping in" point, where you suggested this be done as a > crawler action. Yep, sorry, I neglected to explain my original intent which was Crawler action *and* 1-2 above. > > I can see how we can easily plug in a 1-1 transform action into the crawler, > but I don't see how we can simply do the 150-1, especially if we need to > build in all the requisite clean-up, tracking, and error and off-nominal > condition handling to achieve a reliable system. I think if you implement a NoOp catalog from 1 above and have a separate FM instance, configuring the CrawlerAction that runs on postIngestSuccess to ingest the other 150 files into some other location would be simply giving it the "shadow" FM instance URL. > > I personally believe we haven't gotten to the point that we've justified the > need that the above file reduction (if truly needed for science) be done > pre-ingest, rather than through introducing another PGE workflow within the > PCS, which we understand how to do. Yep that's another way to do it. > > But again, if anyone has a good idea on this, we'd like to hear it. > > > More detailed background: > > 1) This thread started because we've recently learned our primary data > provider is going to provide many (but not all) of our desired file types at > 32 sec, instead of 8 minute, temporal granularity, which was our original > design point. > > 2) Based on a round of interface testing with the data provider a few months > ago, the concern was raised that we may not be able to keep up with the > incoming data flow. Based on earlier testing, the main culprit seems to be > the crawler's ability to keep up with the sheer number of files as opposed > to the data volume. Do you have some #s on this, would be great to share? Can you explain what not being able to "keep up" entails? > > Without going into the details, Brian did make changes to push-pull to tidy > up the directory structure meant to address this problem. So whether this > will still be a issue or not is now a matter of speculation [and we as a > team are quite good at speculation]. :-) > > The obvious next step is to try to repeat the test (or simulate it as best > we can with ourselves as the data provider). I believe there is a space of > relatively simple optimizations here that we have yet to explore (for > example, running multiple crawlers). Yep that's one way to do it too, just run another crawler (or set of them) to move or cleanup those files. > > 3) A related concern is whether the system can handle a ~20 fold increase in > the total number of files in the file catalog. An underling issue from the > very beginning of this project is whether we should design the system based > on the fundamental instrument data granularity (32 sec) or a larger > "aggregation" granularity based on our choosing (8 min). This is because > upstream data provider has always advertised that they could provide all > customers with aggregated products at any granularity that was desired. But > based our understanding of their system, we always had a lingering concern > about their ability (and reliability) to do this per our specifications. > Some of us argued that if the system could handle it, we should just adopt > the naïve 32 sec granularity across the board for all the incoming data, > because then everything would be under our control. > > Nevertheless, because of earlier assurances from our data provider, and > because we haven't ever had numbers demonstrating PCS performance for high > file loading, and because of science team aversion to the smaller 32 sec > file size, we designed the system to 8 min. And now the data provider is > partially back tracking. > > Before he left, Brian worked with our DBA to redesign the file manager > backend to store file metadata as explicit DB typed data columns (rather > than through Strings referenced generically through name, value). So this > should give us significant growth potential in terms of DB/filemgr > performance. Thus there is reason to believe system scaling concern will no > longer be an issue. But we don't know how we can be sure. Best way to do is test! Also would love to see the ColumnBasedCatalog that Brian worked on that's in a branch be backported to trunk. Any takers on your end? :) > > Of course, if number of files really proves to really be a problem, we > always have the relatively straight-forward option of having the operator > (or some process) routinely delete (or un-catalog) the 32 sec files when > we're done aggregating them. +1 > > 4) Because of above concerns, one solution put forward is that we NEVER > ingest the 32 sec files into the file catalog, but rather do the 150-1 > aggregation before ingestion. This is where you suggested a crawler > action... No probs. See my reply above (1, 2, specifically) and let's take it from there. Cheers, Chris ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: [email protected] WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
