[jira] [Commented] (SOLR-12753) Async logging ring buffer and OOM error
[ https://issues.apache.org/jira/browse/SOLR-12753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775605#comment-16775605 ] ASF subversion and git services commented on SOLR-12753: Commit 07cc2d98ef97b78a5161e451ef18ed899f5d7dd3 in lucene-solr's branch refs/heads/branch_8x from erick [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=07cc2d9 ] SOLR-12055: Enable async logging by default SOLR-12753: Async logging ring buffer and OOM error (cherry picked from commit 0de3905) > Async logging ring buffer and OOM error > --- > > Key: SOLR-12753 > URL: https://issues.apache.org/jira/browse/SOLR-12753 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: logging >Affects Versions: 7.5 >Reporter: Andrzej Bialecki >Assignee: Erick Erickson >Priority: Major > Attachments: SOLR-12753.patch > > > I’m using a simulated environment for autoscaling tests, which may create > some pretty degenerate cases (like collections with 50,000 replicas and > Policy calculations over these, times 500 nodes). > I noticed that when I turned on debug logging I occasionally would get an OOM > error, and the heap dump showed that the biggest objects were a bunch of > extremely large strings in the async logger’s ring buffer. These strings were > admittedly extremely large (million chars or so) but the previously used sync > logging didn’t have any issue with them, because they were consumed one by > one. > For sure, Solr should not attempt to be logging multi-megabyte data. But I > also feel like the framework could perhaps help here by enforcing large but > sane limits on maximum size of log messages. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12753) Async logging ring buffer and OOM error
[ https://issues.apache.org/jira/browse/SOLR-12753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775612#comment-16775612 ] ASF subversion and git services commented on SOLR-12753: Commit 4aa0645ea6216f556ffd8c3ad6fcb276a0cc796d in lucene-solr's branch refs/heads/branch_8x from erick [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=4aa0645 ] SOLR-12055: SOLR-12753: Missed mentioning SOLR-12753 in 8.1 CHANGES.txt > Async logging ring buffer and OOM error > --- > > Key: SOLR-12753 > URL: https://issues.apache.org/jira/browse/SOLR-12753 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: logging >Affects Versions: 7.5 >Reporter: Andrzej Bialecki >Assignee: Erick Erickson >Priority: Major > Attachments: SOLR-12753.patch > > > I’m using a simulated environment for autoscaling tests, which may create > some pretty degenerate cases (like collections with 50,000 replicas and > Policy calculations over these, times 500 nodes). > I noticed that when I turned on debug logging I occasionally would get an OOM > error, and the heap dump showed that the biggest objects were a bunch of > extremely large strings in the async logger’s ring buffer. These strings were > admittedly extremely large (million chars or so) but the previously used sync > logging didn’t have any issue with them, because they were consumed one by > one. > For sure, Solr should not attempt to be logging multi-megabyte data. But I > also feel like the framework could perhaps help here by enforcing large but > sane limits on maximum size of log messages. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12753) Async logging ring buffer and OOM error
[ https://issues.apache.org/jira/browse/SOLR-12753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775613#comment-16775613 ] ASF subversion and git services commented on SOLR-12753: Commit 4aa0645ea6216f556ffd8c3ad6fcb276a0cc796d in lucene-solr's branch refs/heads/branch_8x from erick [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=4aa0645 ] SOLR-12055: SOLR-12753: Missed mentioning SOLR-12753 in 8.1 CHANGES.txt > Async logging ring buffer and OOM error > --- > > Key: SOLR-12753 > URL: https://issues.apache.org/jira/browse/SOLR-12753 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: logging >Affects Versions: 7.5 >Reporter: Andrzej Bialecki >Assignee: Erick Erickson >Priority: Major > Attachments: SOLR-12753.patch > > > I’m using a simulated environment for autoscaling tests, which may create > some pretty degenerate cases (like collections with 50,000 replicas and > Policy calculations over these, times 500 nodes). > I noticed that when I turned on debug logging I occasionally would get an OOM > error, and the heap dump showed that the biggest objects were a bunch of > extremely large strings in the async logger’s ring buffer. These strings were > admittedly extremely large (million chars or so) but the previously used sync > logging didn’t have any issue with them, because they were consumed one by > one. > For sure, Solr should not attempt to be logging multi-megabyte data. But I > also feel like the framework could perhaps help here by enforcing large but > sane limits on maximum size of log messages. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12753) Async logging ring buffer and OOM error
[ https://issues.apache.org/jira/browse/SOLR-12753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775607#comment-16775607 ] ASF subversion and git services commented on SOLR-12753: Commit 4530c8041ff0c7b7bf68f2412ce14a3bca2de1db in lucene-solr's branch refs/heads/branch_8x from erick [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=4530c80 ] SOLR-12055: Enable async logging by default SOLR-12753: Async logging ring buffer and OOM error > Async logging ring buffer and OOM error > --- > > Key: SOLR-12753 > URL: https://issues.apache.org/jira/browse/SOLR-12753 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: logging >Affects Versions: 7.5 >Reporter: Andrzej Bialecki >Assignee: Erick Erickson >Priority: Major > Attachments: SOLR-12753.patch > > > I’m using a simulated environment for autoscaling tests, which may create > some pretty degenerate cases (like collections with 50,000 replicas and > Policy calculations over these, times 500 nodes). > I noticed that when I turned on debug logging I occasionally would get an OOM > error, and the heap dump showed that the biggest objects were a bunch of > extremely large strings in the async logger’s ring buffer. These strings were > admittedly extremely large (million chars or so) but the previously used sync > logging didn’t have any issue with them, because they were consumed one by > one. > For sure, Solr should not attempt to be logging multi-megabyte data. But I > also feel like the framework could perhaps help here by enforcing large but > sane limits on maximum size of log messages. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12753) Async logging ring buffer and OOM error
[ https://issues.apache.org/jira/browse/SOLR-12753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16775580#comment-16775580 ] ASF subversion and git services commented on SOLR-12753: Commit 0de3905ce71c53b91b28972fb327d35c6a604e6b in lucene-solr's branch refs/heads/master from Erick Erickson [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=0de3905 ] SOLR-12055: Enable async logging by default SOLR-12753: Async logging ring buffer and OOM error > Async logging ring buffer and OOM error > --- > > Key: SOLR-12753 > URL: https://issues.apache.org/jira/browse/SOLR-12753 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: logging >Affects Versions: 7.5 >Reporter: Andrzej Bialecki >Assignee: Erick Erickson >Priority: Major > Attachments: SOLR-12753.patch > > > I’m using a simulated environment for autoscaling tests, which may create > some pretty degenerate cases (like collections with 50,000 replicas and > Policy calculations over these, times 500 nodes). > I noticed that when I turned on debug logging I occasionally would get an OOM > error, and the heap dump showed that the biggest objects were a bunch of > extremely large strings in the async logger’s ring buffer. These strings were > admittedly extremely large (million chars or so) but the previously used sync > logging didn’t have any issue with them, because they were consumed one by > one. > For sure, Solr should not attempt to be logging multi-megabyte data. But I > also feel like the framework could perhaps help here by enforcing large but > sane limits on maximum size of log messages. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12753) Async logging ring buffer and OOM error
[ https://issues.apache.org/jira/browse/SOLR-12753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701078#comment-16701078 ] Erick Erickson commented on SOLR-12753: --- [~ab]WDYT is a good length for the max size? The current patch tops it off at 10K. Too large? Too small? > Async logging ring buffer and OOM error > --- > > Key: SOLR-12753 > URL: https://issues.apache.org/jira/browse/SOLR-12753 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: logging >Affects Versions: 7.5 >Reporter: Andrzej Bialecki >Assignee: Erick Erickson >Priority: Major > Attachments: SOLR-12753.patch > > > I’m using a simulated environment for autoscaling tests, which may create > some pretty degenerate cases (like collections with 50,000 replicas and > Policy calculations over these, times 500 nodes). > I noticed that when I turned on debug logging I occasionally would get an OOM > error, and the heap dump showed that the biggest objects were a bunch of > extremely large strings in the async logger’s ring buffer. These strings were > admittedly extremely large (million chars or so) but the previously used sync > logging didn’t have any issue with them, because they were consumed one by > one. > For sure, Solr should not attempt to be logging multi-megabyte data. But I > also feel like the framework could perhaps help here by enforcing large but > sane limits on maximum size of log messages. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12753) Async logging ring buffer and OOM error
[ https://issues.apache.org/jira/browse/SOLR-12753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612627#comment-16612627 ] Erick Erickson commented on SOLR-12753: --- Since I backed out the async logging, I'll probably fold this into 12055 when I check it back in after the 7.5 release. > Async logging ring buffer and OOM error > --- > > Key: SOLR-12753 > URL: https://issues.apache.org/jira/browse/SOLR-12753 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: logging >Affects Versions: 7.5 >Reporter: Andrzej Bialecki >Assignee: Erick Erickson >Priority: Major > Attachments: SOLR-12753.patch > > > I’m using a simulated environment for autoscaling tests, which may create > some pretty degenerate cases (like collections with 50,000 replicas and > Policy calculations over these, times 500 nodes). > I noticed that when I turned on debug logging I occasionally would get an OOM > error, and the heap dump showed that the biggest objects were a bunch of > extremely large strings in the async logger’s ring buffer. These strings were > admittedly extremely large (million chars or so) but the previously used sync > logging didn’t have any issue with them, because they were consumed one by > one. > For sure, Solr should not attempt to be logging multi-megabyte data. But I > also feel like the framework could perhaps help here by enforcing large but > sane limits on maximum size of log messages. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12753) Async logging ring buffer and OOM error
[ https://issues.apache.org/jira/browse/SOLR-12753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607497#comment-16607497 ] Erick Erickson commented on SOLR-12753: --- [~ab] Can you give this a whirl, I've found a pattern layout that limits the length of the message. Notes: * I've arbitrarily limited the length of the messages to 10240. This is a straw-man number, WDYT? * I've assumed that the truncation is done before the message is put in the buffer. I have no proof of that but it seems logical * Truncated messages will have elipses at the end. Interestingly when I made the length extremely short (10) there were no elipses. Let me know if this fixes the problem you saw and anyone who wants to chime in on the default message length please do. Having a default there _does_ allow us to predict the max memory consumption. > Async logging ring buffer and OOM error > --- > > Key: SOLR-12753 > URL: https://issues.apache.org/jira/browse/SOLR-12753 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: logging >Affects Versions: 7.5 >Reporter: Andrzej Bialecki >Assignee: Erick Erickson >Priority: Major > Attachments: SOLR-12753.patch > > > I’m using a simulated environment for autoscaling tests, which may create > some pretty degenerate cases (like collections with 50,000 replicas and > Policy calculations over these, times 500 nodes). > I noticed that when I turned on debug logging I occasionally would get an OOM > error, and the heap dump showed that the biggest objects were a bunch of > extremely large strings in the async logger’s ring buffer. These strings were > admittedly extremely large (million chars or so) but the previously used sync > logging didn’t have any issue with them, because they were consumed one by > one. > For sure, Solr should not attempt to be logging multi-megabyte data. But I > also feel like the framework could perhaps help here by enforcing large but > sane limits on maximum size of log messages. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
