The whole russian analyzer is very strange and works against all charset/unicode conventions. It defines own "charsets" (the only valid one is UNICODE), which are all applied to standard java 16 bit chars. The test shows, how this works: It open a text file in KOI8 using the "ISO-88591-1" charset (just to not modify the codepoints when converting to 16bit java chars (in principle it does a deprecated "new String(byte[],0)"). These completely wrong java chars are then handled by an analyzers's internal charset conversion (working on the 16 bit chars).
The only correct usage of this package is: - open file with correct encoding (when instantiating the Reader, so specify KOI8 or windows1251 to the Reader). The string is then correctly UTF-16 encoded java chars. On this string the "pseudo-charset" UNICODE of this analyzer can be used. In my opinion, this invalid usage of java chars should be deprecated, the only correct pseudo-charset should be the one specified by UNICODE and all charset conversions should be done using the Reader. Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -----Original Message----- > From: Robert Muir [mailto:rcm...@gmail.com] > Sent: Monday, July 06, 2009 3:08 PM > To: java-dev@lucene.apache.org > Subject: Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen() > excessively in IndexReader and IndexWriter > > Uwe, I think so too. This way it will not be prone to breakage again. > > On Mon, Jul 6, 2009 at 8:38 AM, Uwe Schindler<u...@thetaphi.de> wrote: > > In my opinion, these files should be converted to UTF-8 and committed > again > > (and the Reader in the test recondigured for UTF-8). Then they can be > native > > EOL style again. The problem is that SVN can only handle the EOL style > for > > one-byte-per-char and UTF-8 files. > > > > I give it a try here (and I have a converter). > > > > ----- > > Uwe Schindler > > H.-H.-Meier-Allee 63, D-28213 Bremen > > http://www.thetaphi.de > > eMail: u...@thetaphi.de > > > >> -----Original Message----- > >> From: Robert Muir [mailto:rcm...@gmail.com] > >> Sent: Monday, July 06, 2009 1:11 PM > >> To: java-dev@lucene.apache.org > >> Subject: Re: [jira] Commented: (LUCENE-1707) Don't use ensureOpen() > >> excessively in IndexReader and IndexWriter > >> > >> yeah, its fixed now. > >> > >> On Mon, Jul 6, 2009 at 7:06 AM, Michael > >> McCandless<luc...@mikemccandless.com> wrote: > >> > Is this the native vs LF svn:eol-style that Uwe already fixed? > >> > > >> > Mike > >> > > >> > On Thu, Jul 2, 2009 at 10:03 AM, Shai Erera<ser...@gmail.com> wrote: > >> >> Can somebody try to revert the change and test it on Windows? > >> >> > >> >> On Thu, Jul 2, 2009 at 4:44 PM, Robert Muir <rcm...@gmail.com> > wrote: > >> >>> > >> >>> well then I have no idea why it doesn't fail. Except that perhaps > its > >> >>> EOL-related (as Shai said), and that the failure is somehow > >> >>> platform-dependent due to newline differences between windows and > unix > >> >>> (and the way these are encoded in UTF-16/stored in SVN)? > >> >>> > >> >>> I don't do really any work with files in UTF-16 so this is just a > >> theory. > >> >>> > >> >>> On Thu, Jul 2, 2009 at 9:40 AM, Mark Miller<markrmil...@gmail.com> > >> wrote: > >> >>> > Hudson runs all the tests and emails java-dev if any of them > fail. > >> >>> > > >> >>> > On Thu, Jul 2, 2009 at 9:37 AM, Robert Muir (JIRA) > <j...@apache.org> > >> >>> > wrote: > >> >>> >> > >> >>> >> [ > >> >>> >> > >> >>> >> https://issues.apache.org/jira/browse/LUCENE- > >> 1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment- > >> tabpanel&focusedCommentId=12726479#action_12726479 > >> >>> >> ] > >> >>> >> > >> >>> >> Robert Muir commented on LUCENE-1707: > >> >>> >> ------------------------------------- > >> >>> >> > >> >>> >> bq. Why doesn't Hudson encounter this problem? > >> >>> >> > >> >>> >> Forgive my ignorance, does hudson also run tests or just verify > >> build? > >> >>> >> These files are only used in tests! > >> >>> >> > >> >>> >> I agree we should correct it, and perhaps to prevent other > problems > >> >>> >> these > >> >>> >> files should be converted to UTF-8. > >> >>> >> > >> >>> >> For the record I am still confused about these java-code > analyzers > >> that > >> >>> >> implement snowball algorithms, why do they exist when the same > >> >>> >> functionality > >> >>> >> is in contrib/snowball? > >> >>> >> > >> >>> >> > >> >>> >> > Don't use ensureOpen() excessively in IndexReader and > IndexWriter > >> >>> >> > -------------------------------------------------------------- > --- > >> >>> >> > > >> >>> >> > Key: LUCENE-1707 > >> >>> >> > URL: > >> >>> >> > https://issues.apache.org/jira/browse/LUCENE-1707 > >> >>> >> > Project: Lucene - Java > >> >>> >> > Issue Type: Improvement > >> >>> >> > Components: Index > >> >>> >> > Reporter: Shai Erera > >> >>> >> > Fix For: 2.9 > >> >>> >> > > >> >>> >> > Attachments: LUCENE-1707.patch, LUCENE-1707.patch > >> >>> >> > > >> >>> >> > > >> >>> >> > A spin off from here: > >> >>> >> > http://www.nabble.com/Excessive-use-of-ensureOpen()- > >> td24127806.html. > >> >>> >> > We should stop calling this method when it's not necessary for > >> any > >> >>> >> > internal Lucene code. Currently, this code seems to hurt > properly > >> >>> >> > written > >> >>> >> > apps, unnecessarily. > >> >>> >> > Will post a patch soon > >> >>> >> > >> >>> >> -- > >> >>> >> This message is automatically generated by JIRA. > >> >>> >> - > >> >>> >> You can reply to this email to add a comment to the issue > online. > >> >>> >> > >> >>> >> > >> >>> >> ---------------------------------------------------------------- > --- > >> -- > >> >>> >> To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org > >> >>> >> For additional commands, e-mail: java-dev-h...@lucene.apache.org > >> >>> >> > >> >>> > > >> >>> > > >> >>> > > >> >>> > -- > >> >>> > -- > >> >>> > - Mark > >> >>> > > >> >>> > http://www.lucidimagination.com > >> >>> > > >> >>> > > >> >>> > >> >>> > >> >>> > >> >>> -- > >> >>> Robert Muir > >> >>> rcm...@gmail.com > >> >>> > >> >>> ------------------------------------------------------------------- > -- > >> >>> To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org > >> >>> For additional commands, e-mail: java-dev-h...@lucene.apache.org > >> >>> > >> >> > >> >> > >> > > >> > --------------------------------------------------------------------- > >> > To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org > >> > For additional commands, e-mail: java-dev-h...@lucene.apache.org > >> > > >> > > >> > >> > >> > >> -- > >> Robert Muir > >> rcm...@gmail.com > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org > >> For additional commands, e-mail: java-dev-h...@lucene.apache.org > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org > > For additional commands, e-mail: java-dev-h...@lucene.apache.org > > > > > > > > -- > Robert Muir > rcm...@gmail.com > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-dev-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
