[
https://issues.apache.org/jira/browse/LUCENE-7525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15942446#comment-15942446
]
Michael Braun commented on LUCENE-7525:
---------------------------------------
Should a new utility Char-To-Int map be created in lucene-core utils to store
the char->int (combined chars) map?
> ASCIIFoldingFilter.foldToASCII performance issue due to large compiled method
> size
> ----------------------------------------------------------------------------------
>
> Key: LUCENE-7525
> URL: https://issues.apache.org/jira/browse/LUCENE-7525
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/analysis
> Affects Versions: 6.2.1
> Reporter: Karl von Randow
> Attachments: ASCIIFoldingFilter.java, ASCIIFolding.java,
> LUCENE-7525.patch, LUCENE-7525.patch, TestASCIIFolding.java
>
>
> The {{ASCIIFoldingFilter.foldToASCII}} method has an enormous switch
> statement and is too large for the HotSpot compiler to compile; causing a
> performance problem.
> The method is about 13K compiled, versus the 8KB HotSpot limit. So splitting
> the method in half works around the problem.
> In my tests splitting the method in half resulted in a 5X performance
> increase.
> In the test code below you can see how slow the fold method is, even when it
> is using the shortcut when the character is less than 0x80, compared to an
> inline implementation of the same shortcut.
> So a workaround is to split the method. I'm happy to provide a patch. It's a
> hack, of course. Perhaps using the {{MappingCharFilterFactory}} with an input
> file as per SOLR-2013 would be a better replacement for this method in this
> class?
> {code:java}
> public class ASCIIFoldingFilterPerformanceTest {
> private static final int ITERATIONS = 1_000_000;
> @Test
> public void testFoldShortString() {
> char[] input = "testing".toCharArray();
> char[] output = new char[input.length * 4];
> for (int i = 0; i < ITERATIONS; i++) {
> ASCIIFoldingFilter.foldToASCII(input, 0, output, 0,
> input.length);
> }
> }
> @Test
> public void testFoldShortAccentedString() {
> char[] input = "éúéúøßüäéúéúøßüä".toCharArray();
> char[] output = new char[input.length * 4];
> for (int i = 0; i < ITERATIONS; i++) {
> ASCIIFoldingFilter.foldToASCII(input, 0, output, 0,
> input.length);
> }
> }
> @Test
> public void testManualFoldTinyString() {
> char[] input = "t".toCharArray();
> char[] output = new char[input.length * 4];
> for (int i = 0; i < ITERATIONS; i++) {
> int k = 0;
> for (int j = 0; j < 1; ++j) {
> final char c = input[j];
> if (c < '\u0080') {
> output[k++] = c;
> } else {
> Assert.assertTrue(false);
> }
> }
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
