[jira] [Commented] (LUCENE-4462) Publishing flushed segments is single threaded and too costly
[ https://issues.apache.org/jira/browse/LUCENE-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13610688#comment-13610688 ] Commit Tag Bot commented on LUCENE-4462: [branch_4x commit] Simon Willnauer http://svn.apache.org/viewvc?view=revisionrevision=1397237 LUCENE-4462: Flush Deletes, SegmentInfos and build CFS concurrently in DWPT Publishing flushed segments is single threaded and too costly - Key: LUCENE-4462 URL: https://issues.apache.org/jira/browse/LUCENE-4462 Project: Lucene - Core Issue Type: Improvement Components: core/index Affects Versions: 4.0-ALPHA, 4.0-BETA, 4.0 Reporter: Michael McCandless Assignee: Simon Willnauer Fix For: 4.1, 5.0 Attachments: LUCENE-4462.patch, LUCENE-4462.patch Spinoff from http://lucene.markmail.org/thread/4li6bbomru35qn7w The new TestBagOfPostings failed the build because it timed out after 2 hours ... but in digging I found that it was a starvation issue: the 4 threads were flushing segments much faster than the 1 thread could publish them. I think this is because publishing segments (DocumentsWriter.publishFlushedSegment) is actually rather costly (creates CFS file if necessary, writes .si, etc.). I committed a workaround for now, to prevent starvation (see svn diff -c 1394704 https://svn.apache.org/repos/asf/lucene/dev/trunk), but we really should address the root cause by moving these costly ops into flush() so that publishing is a low cost operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-4462) Publishing flushed segments is single threaded and too costly
[ https://issues.apache.org/jira/browse/LUCENE-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13474445#comment-13474445 ] Simon Willnauer commented on LUCENE-4462: - backported to 4x in revision 1397237. Publishing flushed segments is single threaded and too costly - Key: LUCENE-4462 URL: https://issues.apache.org/jira/browse/LUCENE-4462 Project: Lucene - Core Issue Type: Improvement Components: core/index Affects Versions: 4.0-ALPHA, 4.0-BETA, 4.0 Reporter: Michael McCandless Assignee: Simon Willnauer Fix For: 4.1, 5.0 Attachments: LUCENE-4462.patch, LUCENE-4462.patch Spinoff from http://lucene.markmail.org/thread/4li6bbomru35qn7w The new TestBagOfPostings failed the build because it timed out after 2 hours ... but in digging I found that it was a starvation issue: the 4 threads were flushing segments much faster than the 1 thread could publish them. I think this is because publishing segments (DocumentsWriter.publishFlushedSegment) is actually rather costly (creates CFS file if necessary, writes .si, etc.). I committed a workaround for now, to prevent starvation (see svn diff -c 1394704 https://svn.apache.org/repos/asf/lucene/dev/trunk), but we really should address the root cause by moving these costly ops into flush() so that publishing is a low cost operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-4462) Publishing flushed segments is single threaded and too costly
[ https://issues.apache.org/jira/browse/LUCENE-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13473069#comment-13473069 ] Simon Willnauer commented on LUCENE-4462: - Committed to trunk in revision 1396500 Publishing flushed segments is single threaded and too costly - Key: LUCENE-4462 URL: https://issues.apache.org/jira/browse/LUCENE-4462 Project: Lucene - Core Issue Type: Improvement Components: core/index Affects Versions: 4.0-ALPHA, 4.0-BETA, 4.0 Reporter: Michael McCandless Assignee: Simon Willnauer Fix For: 4.1, 5.0 Attachments: LUCENE-4462.patch, LUCENE-4462.patch Spinoff from http://lucene.markmail.org/thread/4li6bbomru35qn7w The new TestBagOfPostings failed the build because it timed out after 2 hours ... but in digging I found that it was a starvation issue: the 4 threads were flushing segments much faster than the 1 thread could publish them. I think this is because publishing segments (DocumentsWriter.publishFlushedSegment) is actually rather costly (creates CFS file if necessary, writes .si, etc.). I committed a workaround for now, to prevent starvation (see svn diff -c 1394704 https://svn.apache.org/repos/asf/lucene/dev/trunk), but we really should address the root cause by moving these costly ops into flush() so that publishing is a low cost operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-4462) Publishing flushed segments is single threaded and too costly
[ https://issues.apache.org/jira/browse/LUCENE-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13471662#comment-13471662 ] Michael McCandless commented on LUCENE-4462: Patch looks good, thanks Simon! I think we should keep the safety in there (the fallback to forcePurge if too many segments are backlogged)...? Hopefully it never needs to run... but just in case. Publishing flushed segments is single threaded and too costly - Key: LUCENE-4462 URL: https://issues.apache.org/jira/browse/LUCENE-4462 Project: Lucene - Core Issue Type: Improvement Reporter: Michael McCandless Assignee: Simon Willnauer Attachments: LUCENE-4462.patch Spinoff from http://lucene.markmail.org/thread/4li6bbomru35qn7w The new TestBagOfPostings failed the build because it timed out after 2 hours ... but in digging I found that it was a starvation issue: the 4 threads were flushing segments much faster than the 1 thread could publish them. I think this is because publishing segments (DocumentsWriter.publishFlushedSegment) is actually rather costly (creates CFS file if necessary, writes .si, etc.). I committed a workaround for now, to prevent starvation (see svn diff -c 1394704 https://svn.apache.org/repos/asf/lucene/dev/trunk), but we really should address the root cause by moving these costly ops into flush() so that publishing is a low cost operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-4462) Publishing flushed segments is single threaded and too costly
[ https://issues.apache.org/jira/browse/LUCENE-4462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13471882#comment-13471882 ] Simon Willnauer commented on LUCENE-4462: - bq. I think we should keep the safety in there (the fallback to forcePurge if too many segments are backlogged)...? Hopefully it never needs to run... but just in case. I agree, I remove it for beasting. I will add back and commit. I will let this bake in a bit and then port to 4.x Publishing flushed segments is single threaded and too costly - Key: LUCENE-4462 URL: https://issues.apache.org/jira/browse/LUCENE-4462 Project: Lucene - Core Issue Type: Improvement Reporter: Michael McCandless Assignee: Simon Willnauer Attachments: LUCENE-4462.patch Spinoff from http://lucene.markmail.org/thread/4li6bbomru35qn7w The new TestBagOfPostings failed the build because it timed out after 2 hours ... but in digging I found that it was a starvation issue: the 4 threads were flushing segments much faster than the 1 thread could publish them. I think this is because publishing segments (DocumentsWriter.publishFlushedSegment) is actually rather costly (creates CFS file if necessary, writes .si, etc.). I committed a workaround for now, to prevent starvation (see svn diff -c 1394704 https://svn.apache.org/repos/asf/lucene/dev/trunk), but we really should address the root cause by moving these costly ops into flush() so that publishing is a low cost operation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
