Hey Mike Great! Sorry for leading you in the wrong direction.
Cheers, Tom On 18 June 2014 01:21, Mattmann, Chris A (3980) < chris.a.mattm...@jpl.nasa.gov> wrote: > Good job Mike! > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Chris Mattmann, Ph.D. > Chief Architect > Instrument Software and Science Data Systems Section (398) > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA > Office: 168-519, Mailstop: 168-527 > Email: chris.a.mattm...@nasa.gov > WWW: http://sunset.usc.edu/~mattmann/ > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > Adjunct Associate Professor, Computer Science Department > University of Southern California, Los Angeles, CA 90089 USA > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > > > > -----Original Message----- > From: Mike Vogel <mike.vo...@knowledgent.com> > Reply-To: "user@oodt.apache.org" <user@oodt.apache.org> > Date: Tuesday, June 17, 2014 4:19 PM > To: "user@oodt.apache.org" <user@oodt.apache.org> > Subject: RE: Curator UI editing metadata in solr > > >Problem solved. Updates of data from the curator UI were not working > >because of the combination of: > >· > >The setting of org.apache.oodt.cas.filemgr.metadata.expandProduct=true, > >when I set it to false I see the metadata instead of the product data. > >Only > > somewhat clear from reading the code exactly what this property is > >intended to control other than the behavior I encountered. Can anyone > >clarify? > >· > >Filtering out the extra fields that solr is automatically adding as part > >of my ingestion pipeline from the updates that the Curator does, e.g., the > > _version_, and a timestamp field are automatically added but updates of > >them failed. I ignored these fields via a <processor > >class="solr.IgnoreFieldUpdateProcessorFactory"> in solrconfig.xml. I¹ll > >look into the right way to do that via the overrides > > of the metadata set to and from solr in the OODT code. > > > >From: Mike Vogel > > > >Sent: Tuesday, June 17, 2014 3:22 PM > >To: OODT > >Subject: RE: Curator UI editing metadata in solr > > > > > > > >I have the fields defined in solr with a namespace of ³CAS.², see below. > >My other fields don¹t have a namespace, e.g., ProtocolNumber below. I¹m > >not thinking > > its some configuration problem with the policy files so unless something > >obvious has jumped out at someone you should stop thinking about this > >till I make sure it isn¹t something else simple that I¹ve done wrong. > > > >Is there anything written about how to use the groups concept that is > >implemented in the metadata? Is it totally optional or do some features > >depend on it? > > > > <field name="CAS.ProductId" type="string" indexed="true" stored="true" > >required="true" multiValued="false" /> > > <field name="CAS.ProductName" type="string" indexed="true" > >stored="true" required="true" multiValued="false" /> > > <field name="CAS.ProductTypeName" type="string" indexed="true" > >stored="true" required="true" multiValued="false" /> > > <field name="CAS.ProductTypeId" type="string" indexed="true" > >stored="true" required="true" multiValued="false" /> > > <field name="CAS.ProductReceivedTime" type="date" indexed="true" > >stored="true" required="false" multiValued="false" /> > > <field name="CAS.ProductStructure" type="string" indexed="true" > >stored="true" required="false" multiValued="false" /> > > <field name="CAS.ProductTransferStatus" type="string" indexed="true" > >stored="true" required="false" multiValued="false" /> > > > > <field name="CAS.RootReferenceOriginal" type="string" indexed="true" > >stored="true" required="false" multiValued="true" /> > > <field name="CAS.RootReferenceDatastore" type="string" indexed="true" > >stored="true" required="false" multiValued="true" /> > > <field name="CAS.RootReferenceFileSize" type="long" indexed="true" > >stored="true" required="false" multiValued="true" /> > > <field name="CAS.RootReferenceMimeType" type="string" indexed="true" > >stored="true" required="false" multiValued="true" /> > > > > <field name="CAS.ReferenceOriginal" type="string" indexed="true" > >stored="true" required="false" multiValued="true" /> > > <field name="CAS.ReferenceDatastore" type="string" indexed="true" > >stored="true" required="false" multiValued="true" /> > > <field name="CAS.ReferenceFileSize" type="long" indexed="true" > >stored="true" required="false" multiValued="true" /> > > <field name="CAS.ReferenceMimeType" type="string" indexed="true" > >stored="true" required="false" multiValued="true" /> > > > > <field name="ProtocolNumber" type="string" indexed="true" > >stored="true" required="false" multiValued="true" /> > > <field name="AssayPurpose" type="text_en" indexed="true" stored="true" > >required="false" multiValued="true" /> > > <field name="Target" type="text_en" indexed="true" stored="true" > >required="false" multiValued="true" /> > > <field name="Investigator" type="text_en" indexed="true" stored="true" > >required="false" multiValued="true" /> > > <field name="ProjectDescription" type="text_en" indexed="true" > >stored="true" required="false" multiValued="true" /> > > > >From: Thomas Bennett [mailto:lmzxq....@gmail.com] > > > >Sent: Tuesday, June 17, 2014 8:08 AM > >To: OODT > >Subject: Re: Curator UI editing metadata in solr > > > >Hi Mike, > > > > > >Since the solr catalogue persists the CAS namespace protection for core > >metadata values, perhaps the problem you are seeing might be caused by > >this. > > > > > > > >A way for you to test it, would be to, if possible, use the CAS. > >namespace for your core metadata values (i.e ProductId, should be > >CAS.ProductId, etc). > > > > > > > >Question to OODT dev guys - does the curator support this? > > > > > > > > > >Cheers, > > > >Tom > > > > > > > >------ > > > > > > > >PS: Here listed are snippets from three files with important lines > >highlighted in red, to point out this issue for discussion. > > > > > > > >- filemgr/catalog/solr/Parameters.java > > > >- filmegr/catalog/solr/SolrCatalog.java > > > >- filemgr/catalog/solr/DefaultProductSerializer.java > > > > > > > >Starting with the definition of Parameters, in > >filemgr/catalog/solr/Parameters.java > > > > > > > >public class Parameters { > > > > // the Solr unique identifier field > > > > public final static String ID = "id"; > > > > > > > > > >public final static String NS = "CAS."; > > > > > > > > public final static String PRODUCT_ID = NS+"ProductId"; > > > > public final static String PRODUCT_NAME = NS+"ProductName"; > > > > public final static String PRODUCT_STRUCTURE = > >NS+"ProductStructure"; > > > > public final static String PRODUCT_TRANSFER_STATUS = > >NS+"ProductTransferStatus"; > > > > public final static String PRODUCT_RECEIVED_TIME = > >NS+"ProductReceivedTime"; > > > > public final static String PRODUCT_TYPE_NAME = > >NS+"ProductTypeName"; > > > > public final static String PRODUCT_TYPE_ID = NS+"ProductTypeId"; > > > > > > > > > >-------------- > > > > > > > >Here is the addMedata method in filmegr/catalog/solr/SolrCatalog.java > > > > > > > > @Override > > > > public void addMetadata(Metadata metadata, Product product) > >throws CatalogException { > > > > > > > > LOG.info("Adding metadata for > >product:"+product.getProductName()); > > > > > > > > // serialize metadadta to Solr document(s) > > > > // replace=false i.e. add metadata to existing values > > > > List<String> docs = > >productSerializer.serialize(product.getProductId(), metadata, false); > > > > > > > > > >-------------- > > > > > > > >Looking at the appropriate method signature in > >filemgr/catalog/solr/DefaultProductSerializer.java > > > > > > > > /** > > > > * {@inheritDoc} > > > > */ > > > > public List<String> serialize(String productId, Metadata > >metadata, boolean replace) { > > > > > > > > Map<String, List<String>> fields = new HashMap<String, > >List<String>>(); > > > > > > > > for (String key : metadata.getKeys()) { > > > > > >if (! (key.startsWith(Parameters.NS) // skip metadata keys > >starting with reserved namespace > > > > || > >Parameters.PRODUCT_TYPE_NAME.indexOf(key)>=0 // skip 'ProductType' as > >already stored as 'CAS.ProductTypeName' > > > > || > >Parameters.PRODUCT_STRUCTURE.indexOf(key)>=0)) { // skip 'ProductType' as > >already stored as 'CAS.ProductStructure' > > > > for (String value : > >metadata.getAllMetadata(key)) { > > > > this.addKeyValueToMap(fields, > >key, value); > > > > } > > > > } > > > > } > > > > > > > > return this.generateUpdateDocuments(productId, fields, > >replace); > > > > > > > > } > > > > > > > > > > > > > > > > > > > >
