[ http://issues.apache.org/jira/browse/NUTCH-139?page=comments#action_12364112 ]
Jerome Charron commented on NUTCH-139: -------------------------------------- In fact, the more I look at this, the more I agreed with last Doug comment. There is no real needs (for now) for a so complicated meta-info container. I would like to summarize the key goals related to this issue: 1. Defines some constants for protocol and content metadata names. 2. Provides some correction mechanisms for erroneous protocol headers names. 3. Handles multi-valued properties (such as SMTP recipients, or TAGS attached to a html page, ...) 4. Provides a easy way to keep tracks of protocol original values even if they are overridden by parsers (I don't think there a need for a concept of original value at the parsers level. If a parser override a previously existing value setted by another parser, then this new value must replace the existing one). I really think that one of my comment (13/Jan/06 - http://issues.apache.org/jira/browse/NUTCH-139#action_12362618) covers all these cases. In this proposal, the ParseData object keeps a reference on the protocol original metadata map (ContentProperties), instead of copying the map into a new one. The policy is then as follow : * The ContentProperties is created at the protocol level and is then never modified after. * The ParseProperties is created by the content parser and is the place to store any kind of metadata in all the next nutch processes. * Any metadata stored in ParseProperties can be "overridded" (the last who speak has the last word). > Standard metadata property names in the ParseData metadata > ---------------------------------------------------------- > > Key: NUTCH-139 > URL: http://issues.apache.org/jira/browse/NUTCH-139 > Project: Nutch > Type: Improvement > Components: fetcher > Versions: 0.7.1, 0.7, 0.6, 0.7.2-dev, 0.8-dev > Environment: Power Mac OS X 10.4, Dual Processor G5 2.0 Ghz, 1.5 GB RAM, > although bug is independent of environment > Reporter: Chris A. Mattmann > Assignee: Chris A. Mattmann > Priority: Minor > Fix For: 0.7.2-dev, 0.8-dev, 0.7.1, 0.7, 0.6 > Attachments: NUTCH-139.060105.patch, NUTCH-139.Mattmann.patch.txt, > NUTCH-139.jc.review.patch.txt > > Currently, people are free to name their string-based properties anything > that they want, such as having names of "Content-type", "content-TyPe", > "CONTENT_TYPE" all having the same meaning. Stefan G. I believe proposed a > solution in which all property names be converted to lower case, but in > essence this really only fixes half the problem right (the case of > identifying that "CONTENT_TYPE" > and "conTeNT_TyPE" and all the permutations are really the same). What about > if I named it "Content Type", or "ContentType"? > I propose that a way to correct this would be to create a standard set of > named Strings in the ParseData class that the protocol framework and the > parsing framework could use to identify common properties such as > "Content-type", "Creator", "Language", etc. > The properties would be defined at the top of the ParseData class, something > like: > public class ParseData{ > ..... > public static final String CONTENT_TYPE = "content-type"; > public static final String CREATOR = "creator"; > .... > } > In this fashion, users could at least know what the name of the standard > properties that they can obtain from the ParseData are, for example by making > a call to ParseData.getMetadata().get(ParseData.CONTENT_TYPE) to get the > content type or a call to ParseData.getMetadata().set(ParseData.CONTENT_TYPE, > "text/xml"); Of course, this wouldn't preclude users from doing what they are > currently doing, it would just provide a standard method of obtaining some of > the more common, critical metadata without pouring over the code base to > figure out what they are named. > I'll contribute a patch near the end of the this week, or beg. of next week > that addresses this issue. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira ------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
