I know all about formatted text -- I worked at MarkLogic. That is why I mentioned the XML Infoset.
Numeric entities are part of the final presentation, really, part of the encoding. They should never be stored. Always store the Unicode. Numeric and named entities are a convenience for tools and encodings that can't handle Unicode. That is all they are. wunder On Nov 21, 2013, at 9:02 AM, "Jack Krupansky" <j...@basetechnology.com> wrote: > Ah... now I understand your perspective - you have taken a narrow view of > what "text" is. A broader view is that it can contain formatting and special > "entities" as well, or rich text in general. My "read" is that it all depends > on the nature of the application and its requirements, not a "one size fits > all" approach. The four main approaches being pure ASCII, Unicode/UTF-8, SGML > for non-ASCII characters, and full HTML for formatting and rich text. And let > the app needs determine which is most appropriate for each piece of text. > > The goal of SGML and HTML is not to hard-wire the final presentation, but > simply to preserve some level of source format and structure, and then apply > final presentation formatting on top of that. > > Some apps may opt to store the same information in multiple formats, such as > one for raw text search, one for basic display, and one for "detail" display. > > I'm more of a "platform" guy than an "app-specific" guy - give the app > developer tools that they can blend to meet their own requirements (or > interests or tastes.) > > But Solr users should make no mistake, SGML entities are a perfectly valid > intermediate format for rich text. > > -- Jack Krupansky > > -----Original Message----- From: Walter Underwood > Sent: Thursday, November 21, 2013 11:44 AM > To: solr-user@lucene.apache.org > Subject: Re: How to index X™ as ™ (HTML decimal entity) > > And this is the exact problem. Some characters are stored as entities, some > are not. When it is time to display, what else needs escaped? At a minimum, > you would have to always store & as & to avoid escaping the leading > ampersand in the entities. > > You could store every single character as a numeric entity. Or you could > store every non-ASCII character as a numeric entity. Or every non-Latin1 > character. Plus ampersand, of course. > > In these e-mails, we are distinguishing between ™ and ™. How would you > do that? By storing "™" as "&trade;". > > To avoid all this double-think, always store text as Unicode code points, > encoded with a standard Unicode method (UTF-8, etc.). > > When displaying, only make entities if the codepoints cannot be represented > in the target character encoding. If you are sending things in US-ASCII, you > will be sending lots of entities. > > A good encoding library has callbacks for characters that cannot be > represented. You can use these callbacks to format out-of-charset codepoints > as entities. I've done this in product code, it really works. > > Finally, if you don't believe me, believe the XML Infoset, where numeric > entities are always interpreted as treated as Unicode codepoints. > > The other way to go insane is storing local time in the database. Always > store UTC and convert at the edges. > > wunder > > On Nov 21, 2013, at 7:50 AM, "Jack Krupansky" <j...@basetechnology.com> wrote: > >> "Would you store "a" as "A" ?" >> >> No, not in any case. >> >> -- Jack Krupansky >> >> -----Original Message----- From: Michael Sokolov >> Sent: Thursday, November 21, 2013 8:56 AM >> To: solr-user@lucene.apache.org >> Subject: Re: How to index X™ as ™ (HTML decimal entity) >> >> I have to agree w/Walter. Use unicode as a storage format. The entity >> encodings are for transfer/interchange. Encode/decode on the way in and >> out if you have to. Would you store "a" as "A" ? It makes it >> impossible to search for, for one thing. What if someone wants to >> search for the TM character? >> >> -Mike >> >> On 11/20/13 12:07 PM, Jack Krupansky wrote: >>> AFAICT, it's not an "extremely bad idea" - using SGML/HTML as a format for >>> storing text to be rendered. If you disagree - try explaining yourself. >>> >>> But maybe TM should be encoded as "™". Ditto for other named SGML >>> entities. >>> >>> -- Jack Krupansky >>> >>> -----Original Message----- From: Walter Underwood >>> Sent: Wednesday, November 20, 2013 11:21 AM >>> To: solr-user@lucene.apache.org >>> Subject: Re: How to index X™ as ™ (HTML decimal entity) >>> >>> Again, I'd like to know why this is wanted. It sounds like an X-Y, problem. >>> Storing Unicode characters as XML/HTML encoded character references is an >>> extremely bad idea. >>> >>> wunder >>> >>> On Nov 20, 2013, at 5:01 AM, "Jack Krupansky" <j...@basetechnology.com> >>> wrote: >>> >>>> Any analysis filtering affects the indexed value only, but the stored >>>> value would be unchanged from the original input value. An update >>>> processor lets you modify the original input value that will be stored. >>>> >>>> -- Jack Krupansky >>>> >>>> -----Original Message----- From: Uwe Reh >>>> Sent: Wednesday, November 20, 2013 5:43 AM >>>> To: solr-user@lucene.apache.org >>>> Subject: Re: How to index X™ as ™ (HTML decimal entity) >>>> >>>> What's about having a simple charfilter in the analyzer queue for >>>> indexing *and* searching. e.g >>>> <charFilter class="solr.PatternReplaceFilterFactory" pattern="™" >>>> replacement="™" /> >>>> or >>>> <charFilter class="solr.MappingCharFilterFactory" >>>> mapping="mapping-specials.txt" /> >>>> >>>> Uwe >>>> >>>> Am 19.11.2013 23:46, schrieb Developer: >>>>> I have a data coming in to SOLR as below. >>>>> >>>>> <field name="displayName">X™ - Black</field> >>>>> >>>>> I need to store the HTML Entity (decimal) equivalent value (i.e. ™) >>>>> in SOLR rather than storing the original value. >>>>> >>>>> Is there a way to do this? >>>> >>> >>> -- >>> Walter Underwood >>> wun...@wunderwood.org >>> >>> >> > > -- > Walter Underwood > wun...@wunderwood.org > > > -- Walter Underwood wun...@wunderwood.org
