Problem solved. Updates of data from the curator UI were not working because of the combination of:
· The setting of org.apache.oodt.cas.filemgr.metadata.expandProduct=true, when I set it to false I see the metadata instead of the product data. Only somewhat clear from reading the code exactly what this property is intended to control other than the behavior I encountered. Can anyone clarify? · Filtering out the extra fields that solr is automatically adding as part of my ingestion pipeline from the updates that the Curator does, e.g., the _version_, and a timestamp field are automatically added but updates of them failed. I ignored these fields via a <processor class="solr.IgnoreFieldUpdateProcessorFactory"> in solrconfig.xml. I’ll look into the right way to do that via the overrides of the metadata set to and from solr in the OODT code. From: Mike Vogel Sent: Tuesday, June 17, 2014 3:22 PM To: OODT Subject: RE: Curator UI editing metadata in solr I have the fields defined in solr with a namespace of “CAS.”, see below. My other fields don’t have a namespace, e.g., ProtocolNumber below. I’m not thinking its some configuration problem with the policy files so unless something obvious has jumped out at someone you should stop thinking about this till I make sure it isn’t something else simple that I’ve done wrong. Is there anything written about how to use the groups concept that is implemented in the metadata? Is it totally optional or do some features depend on it? <field name="CAS.ProductId" type="string" indexed="true" stored="true" required="true" multiValued="false" /> <field name="CAS.ProductName" type="string" indexed="true" stored="true" required="true" multiValued="false" /> <field name="CAS.ProductTypeName" type="string" indexed="true" stored="true" required="true" multiValued="false" /> <field name="CAS.ProductTypeId" type="string" indexed="true" stored="true" required="true" multiValued="false" /> <field name="CAS.ProductReceivedTime" type="date" indexed="true" stored="true" required="false" multiValued="false" /> <field name="CAS.ProductStructure" type="string" indexed="true" stored="true" required="false" multiValued="false" /> <field name="CAS.ProductTransferStatus" type="string" indexed="true" stored="true" required="false" multiValued="false" /> <field name="CAS.RootReferenceOriginal" type="string" indexed="true" stored="true" required="false" multiValued="true" /> <field name="CAS.RootReferenceDatastore" type="string" indexed="true" stored="true" required="false" multiValued="true" /> <field name="CAS.RootReferenceFileSize" type="long" indexed="true" stored="true" required="false" multiValued="true" /> <field name="CAS.RootReferenceMimeType" type="string" indexed="true" stored="true" required="false" multiValued="true" /> <field name="CAS.ReferenceOriginal" type="string" indexed="true" stored="true" required="false" multiValued="true" /> <field name="CAS.ReferenceDatastore" type="string" indexed="true" stored="true" required="false" multiValued="true" /> <field name="CAS.ReferenceFileSize" type="long" indexed="true" stored="true" required="false" multiValued="true" /> <field name="CAS.ReferenceMimeType" type="string" indexed="true" stored="true" required="false" multiValued="true" /> <field name="ProtocolNumber" type="string" indexed="true" stored="true" required="false" multiValued="true" /> <field name="AssayPurpose" type="text_en" indexed="true" stored="true" required="false" multiValued="true" /> <field name="Target" type="text_en" indexed="true" stored="true" required="false" multiValued="true" /> <field name="Investigator" type="text_en" indexed="true" stored="true" required="false" multiValued="true" /> <field name="ProjectDescription" type="text_en" indexed="true" stored="true" required="false" multiValued="true" /> From: Thomas Bennett [mailto:lmzxq....@gmail.com] Sent: Tuesday, June 17, 2014 8:08 AM To: OODT Subject: Re: Curator UI editing metadata in solr Hi Mike, Since the solr catalogue persists the CAS namespace protection for core metadata values, perhaps the problem you are seeing might be caused by this. A way for you to test it, would be to, if possible, use the CAS. namespace for your core metadata values (i.e ProductId, should be CAS.ProductId, etc). Question to OODT dev guys - does the curator support this? Cheers, Tom ------ PS: Here listed are snippets from three files with important lines highlighted in red, to point out this issue for discussion. - filemgr/catalog/solr/Parameters.java - filmegr/catalog/solr/SolrCatalog.java - filemgr/catalog/solr/DefaultProductSerializer.java Starting with the definition of Parameters, in filemgr/catalog/solr/Parameters.java public class Parameters { // the Solr unique identifier field public final static String ID = "id"; public final static String NS = "CAS."; public final static String PRODUCT_ID = NS+"ProductId"; public final static String PRODUCT_NAME = NS+"ProductName"; public final static String PRODUCT_STRUCTURE = NS+"ProductStructure"; public final static String PRODUCT_TRANSFER_STATUS = NS+"ProductTransferStatus"; public final static String PRODUCT_RECEIVED_TIME = NS+"ProductReceivedTime"; public final static String PRODUCT_TYPE_NAME = NS+"ProductTypeName"; public final static String PRODUCT_TYPE_ID = NS+"ProductTypeId"; -------------- Here is the addMedata method in filmegr/catalog/solr/SolrCatalog.java @Override public void addMetadata(Metadata metadata, Product product) throws CatalogException { LOG.info("Adding metadata for product:"+product.getProductName()); // serialize metadadta to Solr document(s) // replace=false i.e. add metadata to existing values List<String> docs = productSerializer.serialize(product.getProductId(), metadata, false); -------------- Looking at the appropriate method signature in filemgr/catalog/solr/DefaultProductSerializer.java /** * {@inheritDoc} */ public List<String> serialize(String productId, Metadata metadata, boolean replace) { Map<String, List<String>> fields = new HashMap<String, List<String>>(); for (String key : metadata.getKeys()) { if (! (key.startsWith(Parameters.NS) // skip metadata keys starting with reserved namespace || Parameters.PRODUCT_TYPE_NAME.indexOf(key)>=0 // skip 'ProductType' as already stored as 'CAS.ProductTypeName' || Parameters.PRODUCT_STRUCTURE.indexOf(key)>=0)) { // skip 'ProductType' as already stored as 'CAS.ProductStructure' for (String value : metadata.getAllMetadata(key)) { this.addKeyValueToMap(fields, key, value); } } } return this.generateUpdateDocuments(productId, fields, replace); }
