[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting
[ https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876134#comment-16876134 ] ASF subversion and git services commented on LUCENE-8862: - Commit 3a2f1fd19b7ee9783149dadb35fcef5f312c5a3d in lucene-solr's branch refs/heads/branch_8x from Atri Sharma [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=3a2f1fd ] LUCENE-8862: Introduce Collector Level Memory Accounting (#729) This commit introduces a mechanism to track the dynamic memory utilization of Collectors and allow setting collector level limits for memory usage. A new bitset collecting Collector is introduced which supports the memory tracker functionality > Collector Level Dynamic Memory Accounting > - > > Key: LUCENE-8862 > URL: https://issues.apache.org/jira/browse/LUCENE-8862 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > Inspired from LUCENE-8855, I am thinking of adding a new interface which > tracks dynamic memory used by Collectors. This shall allow users to get an > accountability as to the memory usage of their Collectors and better plan > their resource capacity. This shall also allow us to add Collector level > limits for memory usage, thus allowing users a finer control over their > resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting
[ https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876130#comment-16876130 ] Atri Sharma commented on LUCENE-8862: - [~jpountz] Thanks for pushing and reviewing! > Collector Level Dynamic Memory Accounting > - > > Key: LUCENE-8862 > URL: https://issues.apache.org/jira/browse/LUCENE-8862 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Time Spent: 3h 20m > Remaining Estimate: 0h > > Inspired from LUCENE-8855, I am thinking of adding a new interface which > tracks dynamic memory used by Collectors. This shall allow users to get an > accountability as to the memory usage of their Collectors and better plan > their resource capacity. This shall also allow us to add Collector level > limits for memory usage, thus allowing users a finer control over their > resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting
[ https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876129#comment-16876129 ] ASF subversion and git services commented on LUCENE-8862: - Commit 6799ca6efde89aefd1f74f82a728c2be5d73ee6f in lucene-solr's branch refs/heads/master from Atri Sharma [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=6799ca6 ] LUCENE-8862: Introduce Collector Level Memory Accounting (#729) This commit introduces a mechanism to track the dynamic memory utilization of Collectors and allow setting collector level limits for memory usage. A new bitset collecting Collector is introduced which supports the memory tracker functionality > Collector Level Dynamic Memory Accounting > - > > Key: LUCENE-8862 > URL: https://issues.apache.org/jira/browse/LUCENE-8862 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > Inspired from LUCENE-8855, I am thinking of adding a new interface which > tracks dynamic memory used by Collectors. This shall allow users to get an > accountability as to the memory usage of their Collectors and better plan > their resource capacity. This shall also allow us to add Collector level > limits for memory usage, thus allowing users a finer control over their > resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting
[ https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874013#comment-16874013 ] Atri Sharma commented on LUCENE-8862: - Updated the PR with latest comments and moved to misc module. Happy to iterate further. > Collector Level Dynamic Memory Accounting > - > > Key: LUCENE-8862 > URL: https://issues.apache.org/jira/browse/LUCENE-8862 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > Inspired from LUCENE-8855, I am thinking of adding a new interface which > tracks dynamic memory used by Collectors. This shall allow users to get an > accountability as to the memory usage of their Collectors and better plan > their resource capacity. This shall also allow us to add Collector level > limits for memory usage, thus allowing users a finer control over their > resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting
[ https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867325#comment-16867325 ] Atri Sharma commented on LUCENE-8862: - I have opened a PR for the same. Please let me know if this looks fine. Once we merge this, I am planning to open a Jira to enable Solr's facet collector to account for memory. For default cases, the limit can be long.MAX_VALUE. Thoughts? > Collector Level Dynamic Memory Accounting > - > > Key: LUCENE-8862 > URL: https://issues.apache.org/jira/browse/LUCENE-8862 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Inspired from LUCENE-8855, I am thinking of adding a new interface which > tracks dynamic memory used by Collectors. This shall allow users to get an > accountability as to the memory usage of their Collectors and better plan > their resource capacity. This shall also allow us to add Collector level > limits for memory usage, thus allowing users a finer control over their > resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting
[ https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866467#comment-16866467 ] Atri Sharma commented on LUCENE-8862: - Right, agreed. Another thought was if we should add the capability to check after collecting N docs, where N is configurable. For heavy Collectors with total hits tending to large, that should still be a valid range. However, it does have a window of error > Collector Level Dynamic Memory Accounting > - > > Key: LUCENE-8862 > URL: https://issues.apache.org/jira/browse/LUCENE-8862 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > > Inspired from LUCENE-8855, I am thinking of adding a new interface which > tracks dynamic memory used by Collectors. This shall allow users to get an > accountability as to the memory usage of their Collectors and better plan > their resource capacity. This shall also allow us to add Collector level > limits for memory usage, thus allowing users a finer control over their > resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting
[ https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866460#comment-16866460 ] Adrien Grand commented on LUCENE-8862: -- Right, but the benefit of doing it from within the collector means that you can move the check to where the allocation happens, as opposed to treating the collector like a black box and having to periodically check memory usage. It would work as well with minimal overhead if allocations occurred in collect(). > Collector Level Dynamic Memory Accounting > - > > Key: LUCENE-8862 > URL: https://issues.apache.org/jira/browse/LUCENE-8862 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > > Inspired from LUCENE-8855, I am thinking of adding a new interface which > tracks dynamic memory used by Collectors. This shall allow users to get an > accountability as to the memory usage of their Collectors and better plan > their resource capacity. This shall also allow us to add Collector level > limits for memory usage, thus allowing users a finer control over their > resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting
[
https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866452#comment-16866452
]
Atri Sharma commented on LUCENE-8862:
-
{quote}doing it in setNextReader might perform the check long after the
allocation actually happened
{quote}
Isnt that what the example code snippet above is doing, unless I am reading it
wrong?
> Collector Level Dynamic Memory Accounting
> -
>
> Key: LUCENE-8862
> URL: https://issues.apache.org/jira/browse/LUCENE-8862
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Atri Sharma
>Priority: Major
>
> Inspired from LUCENE-8855, I am thinking of adding a new interface which
> tracks dynamic memory used by Collectors. This shall allow users to get an
> accountability as to the memory usage of their Collectors and better plan
> their resource capacity. This shall also allow us to add Collector level
> limits for memory usage, thus allowing users a finer control over their
> resources.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting
[ https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866447#comment-16866447 ] Adrien Grand commented on LUCENE-8862: -- bq. your thought is that the memory tracker itself have a limit Yes, my feeling is that it is difficult to decide when to check memory usage otherwise. Doing it in every collect() call would make things slower and doing it in setNextReader might perform the check long after the allocation actually happened (if it happened in collect() for instance)? > Collector Level Dynamic Memory Accounting > - > > Key: LUCENE-8862 > URL: https://issues.apache.org/jira/browse/LUCENE-8862 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > > Inspired from LUCENE-8855, I am thinking of adding a new interface which > tracks dynamic memory used by Collectors. This shall allow users to get an > accountability as to the memory usage of their Collectors and better plan > their resource capacity. This shall also allow us to add Collector level > limits for memory usage, thus allowing users a finer control over their > resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting
[ https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866438#comment-16866438 ] Atri Sharma commented on LUCENE-8862: - Ah, hmm, so your thought is that the memory tracker itself have a limit? That is a good idea. The only caveat I could think of in this scenario would be for CollectorManagers i.e. does each new Collector get its own Tracker, or is the Tracker global to all Collectors? But I guess that is CollectorManager dependent i.e. does it want per thread limit, or a global limit... > Collector Level Dynamic Memory Accounting > - > > Key: LUCENE-8862 > URL: https://issues.apache.org/jira/browse/LUCENE-8862 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > > Inspired from LUCENE-8855, I am thinking of adding a new interface which > tracks dynamic memory used by Collectors. This shall allow users to get an > accountability as to the memory usage of their Collectors and better plan > their resource capacity. This shall also allow us to add Collector level > limits for memory usage, thus allowing users a finer control over their > resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting
[
https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866436#comment-16866436
]
Adrien Grand commented on LUCENE-8862:
--
Could you pass the MemoryTrackerObject that you mentioned above at construction
time? I have something like that in mind:
{code:java}
class BitSetCollector extends SimpleCollector {
final MemoryTrackerObject tracker;
FixedBitSet bitSet = new FixedBitSet(0);
int length = 0;
int docBase = 0;
BitSetCollector(MemoryTrackerObject tracker) {
this.tracker = tracker;
tracker.addAndGet(bitSet.ramBytesUsed());
}
BitSet getBits() {
return bitSet;
}
void doSetNextReader(LeafReaderContext context) throws IOException {
length += context.reader().maxDoc();
FixedBitSet newBitSet = FixedBitSet.ensureCapacity(bitSet, length);
if (newBitSet != bitSet) {
tracker.addAndGet(newBitSet.ramBytesUsed() - bitSet.ramBytesUsed()); //
ideally we'd do that before actually allocating
bitSet = newBitSet;
}
}
void collect(int doc) {
bitSet.set(docBase + doc);
}
}
long perCollectorMemoryLimit = 5 * 1024L * 1024L; // 5MB
MemoryTrackerObject tracker = new MemoryTrackerObject() {
long ramBytesUsed = 0;
long addAndGet(long delta) {
long newRamBytesUsed = ramBytesUsed + delta;
if (newRamBytesUsed > limit) {
throw new Exception();
}
ramBytesUsed = newRamBytesUsed;
return ramBytesUsed;
}
};
BitSetCollector bitSetCollector = new BitSetCollector(tracker);
TotalHitCountCollector hitCountCollector = new TotalHitCountCollector();
searcher.search(new TermQuery(new Term("foo", "bar")),
MultiCollector.wrap(hitCountCollector, bitSetCollector));
int hitCount = hitCountCollector.getTotalHits();
BitSet hits = bitSetCollector.getBits();{code}
> Collector Level Dynamic Memory Accounting
> -
>
> Key: LUCENE-8862
> URL: https://issues.apache.org/jira/browse/LUCENE-8862
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Atri Sharma
>Priority: Major
>
> Inspired from LUCENE-8855, I am thinking of adding a new interface which
> tracks dynamic memory used by Collectors. This shall allow users to get an
> accountability as to the memory usage of their Collectors and better plan
> their resource capacity. This shall also allow us to add Collector level
> limits for memory usage, thus allowing users a finer control over their
> resources.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting
[ https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866431#comment-16866431 ] Atri Sharma commented on LUCENE-8862: - Another option could be to introduce an interface which exposes consumeMemoryUsage and let interested Collectors implement the same, while having a parent Collector which tracks memory using the shared object. WDYT? > Collector Level Dynamic Memory Accounting > - > > Key: LUCENE-8862 > URL: https://issues.apache.org/jira/browse/LUCENE-8862 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > > Inspired from LUCENE-8855, I am thinking of adding a new interface which > tracks dynamic memory used by Collectors. This shall allow users to get an > accountability as to the memory usage of their Collectors and better plan > their resource capacity. This shall also allow us to add Collector level > limits for memory usage, thus allowing users a finer control over their > resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting
[ https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866423#comment-16866423 ] Atri Sharma commented on LUCENE-8862: - [~jpountz] What are your thoughts on the options of implementing this approach, then? The way I see it, we could either add a default method to Collector interface to accept the memory accounting object and let the Collector implementation deal with how it wants to account for memory consumption. The other way is to add the introduce the visitor API, but I agree that it has its caveats. The objective is to have a generic collector-level interface that can be used to build constructs for monitoring and limiting memory usage per query. Implementing Accountable per Collector will not give us that flexibility without a couple of instanceof checks, IIUC. Thoughts? > Collector Level Dynamic Memory Accounting > - > > Key: LUCENE-8862 > URL: https://issues.apache.org/jira/browse/LUCENE-8862 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > > Inspired from LUCENE-8855, I am thinking of adding a new interface which > tracks dynamic memory used by Collectors. This shall allow users to get an > accountability as to the memory usage of their Collectors and better plan > their resource capacity. This shall also allow us to add Collector level > limits for memory usage, thus allowing users a finer control over their > resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting
[ https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866364#comment-16866364 ] Adrien Grand commented on LUCENE-8862: -- I'd like to avoid adding new methods to the collector API for this. I don't think the collector API needs a visitor API since there aren't many use-cases for collecting multiple collectors at once via a MultiCollector, especially since Lucene 8 and top-hits retrieval optimizations: if you want to retrieve both top hits and facets, it is likely a better idea now to run the query twice concurrently, once with a top-hits collector and another time with a faceting collector. This will enable the UI to start displaying top hits while facets are still being computed. Furthermore exposing internals of collectors via a visitor API might prevent some optimizations or the contract would be a bit weird. For instance MultiLeafCollector removes its sub collectors when they throw a CollectionTerminatedException, should we visit these sub collectors or not? We have a similar problem with the Scorer#getChildren API and the BooleanScorer optimization, which makes the Scorer#getChildren API hard to rely on since it is not consistently implemented. > Collector Level Dynamic Memory Accounting > - > > Key: LUCENE-8862 > URL: https://issues.apache.org/jira/browse/LUCENE-8862 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > > Inspired from LUCENE-8855, I am thinking of adding a new interface which > tracks dynamic memory used by Collectors. This shall allow users to get an > accountability as to the memory usage of their Collectors and better plan > their resource capacity. This shall also allow us to add Collector level > limits for memory usage, thus allowing users a finer control over their > resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting
[ https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866349#comment-16866349 ] Atri Sharma commented on LUCENE-8862: - We should probably add a visitLeafCollector(LeafCollector) in the visitor API as well, along with a consumeTerms and consumeMemoryTracker(MemoryTrackerObject). Thoughts? > Collector Level Dynamic Memory Accounting > - > > Key: LUCENE-8862 > URL: https://issues.apache.org/jira/browse/LUCENE-8862 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > > Inspired from LUCENE-8855, I am thinking of adding a new interface which > tracks dynamic memory used by Collectors. This shall allow users to get an > accountability as to the memory usage of their Collectors and better plan > their resource capacity. This shall also allow us to add Collector level > limits for memory usage, thus allowing users a finer control over their > resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting
[ https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865883#comment-16865883 ] Atri Sharma commented on LUCENE-8862: - The reason I was thinking of a visitor api is since it gives a cleaner interface (not all children collectors need to expose the new constructor). Also, having the Visitor API gives the scope of more functionalities that the user could cook up (for eg, something like consumeTerms?) > Collector Level Dynamic Memory Accounting > - > > Key: LUCENE-8862 > URL: https://issues.apache.org/jira/browse/LUCENE-8862 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > > Inspired from LUCENE-8855, I am thinking of adding a new interface which > tracks dynamic memory used by Collectors. This shall allow users to get an > accountability as to the memory usage of their Collectors and better plan > their resource capacity. This shall also allow us to add Collector level > limits for memory usage, thus allowing users a finer control over their > resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting
[ https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865879#comment-16865879 ] Adrien Grand commented on LUCENE-8862: -- I see. Do we need a visitor at all, we could pass the counter to the constructor of the collector? > Collector Level Dynamic Memory Accounting > - > > Key: LUCENE-8862 > URL: https://issues.apache.org/jira/browse/LUCENE-8862 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > > Inspired from LUCENE-8855, I am thinking of adding a new interface which > tracks dynamic memory used by Collectors. This shall allow users to get an > accountability as to the memory usage of their Collectors and better plan > their resource capacity. This shall also allow us to add Collector level > limits for memory usage, thus allowing users a finer control over their > resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting
[ https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865873#comment-16865873 ] Atri Sharma commented on LUCENE-8862: - Visitor could have an API to take in a memory tracking object (maybe as simple as an AtomicLong with some decorations). The patent collector passes it to down, and every supporting collector accepts that visitor and stores the object to update memory usage. Since the object is shared, it should be up to date at any point in time. Of course, we will have to make the updates and reads concurrent safe. Thoughts? > Collector Level Dynamic Memory Accounting > - > > Key: LUCENE-8862 > URL: https://issues.apache.org/jira/browse/LUCENE-8862 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > > Inspired from LUCENE-8855, I am thinking of adding a new interface which > tracks dynamic memory used by Collectors. This shall allow users to get an > accountability as to the memory usage of their Collectors and better plan > their resource capacity. This shall also allow us to add Collector level > limits for memory usage, thus allowing users a finer control over their > resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting
[ https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865859#comment-16865859 ] Adrien Grand commented on LUCENE-8862: -- One you have a visitor API, how would you implement per-collector limits for memory usage? > Collector Level Dynamic Memory Accounting > - > > Key: LUCENE-8862 > URL: https://issues.apache.org/jira/browse/LUCENE-8862 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > > Inspired from LUCENE-8855, I am thinking of adding a new interface which > tracks dynamic memory used by Collectors. This shall allow users to get an > accountability as to the memory usage of their Collectors and better plan > their resource capacity. This shall also allow us to add Collector level > limits for memory usage, thus allowing users a finer control over their > resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting
[ https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865809#comment-16865809 ] Atri Sharma commented on LUCENE-8862: - Thinking more about this, I am more inclined to do the Visitor approach for Collectors, since that gives us more functionality in terms of what can be done with the visitor, and memory accounting can be one aspect where it is used. Unless there are objections, I am planning to work on a patch for initial review. > Collector Level Dynamic Memory Accounting > - > > Key: LUCENE-8862 > URL: https://issues.apache.org/jira/browse/LUCENE-8862 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > > Inspired from LUCENE-8855, I am thinking of adding a new interface which > tracks dynamic memory used by Collectors. This shall allow users to get an > accountability as to the memory usage of their Collectors and better plan > their resource capacity. This shall also allow us to add Collector level > limits for memory usage, thus allowing users a finer control over their > resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting
[
https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865610#comment-16865610
]
Atri Sharma commented on LUCENE-8862:
-
{quote}Memory accounting has a cost API-wise
{quote}
Agreed, I was thinking of adding it as a default method which is essentially a
no-op.
{quote}The only collectors I have in mind that might allocate lots of memory
are the ones that are used to collect matches in a bitset
{quote}
Yeah, that is true. However, my bigger objective is to allow a generic
interface for consuming and using this data (wherever it is exposed). For eg, I
was thinking of adding a Collector which can wrap other Collectors and provide
a way to fence memory usage.
Other way to think about this is if we should add a similar mechanism like
QueryVisitor to Collectors, and then pursue an effort to add memory collection
heuristics to both QueryVisitor and the Collector visitor. Having the Collector
visitor interface will provide ways to have deeper functionality than just the
memory accounting. WDYT?
> Collector Level Dynamic Memory Accounting
> -
>
> Key: LUCENE-8862
> URL: https://issues.apache.org/jira/browse/LUCENE-8862
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Atri Sharma
>Priority: Major
>
> Inspired from LUCENE-8855, I am thinking of adding a new interface which
> tracks dynamic memory used by Collectors. This shall allow users to get an
> accountability as to the memory usage of their Collectors and better plan
> their resource capacity. This shall also allow us to add Collector level
> limits for memory usage, thus allowing users a finer control over their
> resources.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting
[ https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865469#comment-16865469 ] Adrien Grand commented on LUCENE-8862: -- Memory accounting has a cost API-wise. Like queries, I don't think we should change the Collector API to enable memory accounting. The only collectors I have in mind that might allocate lots of memory are the ones that are used to collect matches in a bitset (there is one for facets and another one in Solr if I'm not mistaken, different impl but similar idea). If this proves to be a source of memory issues, we can work on solutions that are specific to those, similarly to how we only added the Accountable interface to TermInSetQuery? > Collector Level Dynamic Memory Accounting > - > > Key: LUCENE-8862 > URL: https://issues.apache.org/jira/browse/LUCENE-8862 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > > Inspired from LUCENE-8855, I am thinking of adding a new interface which > tracks dynamic memory used by Collectors. This shall allow users to get an > accountability as to the memory usage of their Collectors and better plan > their resource capacity. This shall also allow us to add Collector level > limits for memory usage, thus allowing users a finer control over their > resources. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
