Issac, Do you have suggestions on this. I see you were engaged in conversation on the referred PR.
Regards, Abhishek On Thu, Sep 28, 2017 at 5:26 PM, Gurjinder Singh Rathore < [email protected]> wrote: > Hi, > > I'm using gobblin (embedded) in my project to transfer loads of over 10 > million+ rows at a time (single table). But when I run the load, gobbling > starts giving me errors since looks like the number of Extractor instances > grows almost linearly with the number of rows in my source table. This > results into too many connections being opened and at some point no more > Extractor instances can be created because the DBMS is drained out of all > its available connection limit. This is becoming really painful. > > I dug into the code, and found that I could use the following settings to > limit the number of extractors created simultaenously: > > extract.limit.enabled=true > extract.limit.type=pool > extract.limit.pool.size=10 > > However, TaskContext.java has this check: > > Limiter limiter = DefaultLimiterFactory. > newLimiter(this.taskState); > if (!(limiter instanceof NonRefillableLimiter)) { > throw new IllegalArgumentException("The Limiter used with an > Extractor should be an instance of " > + NonRefillableLimiter.class.getSimpleName()); > } > > This was the end of my hope. I absolutely need to use PoolBasedLimiter > which is not a NonRefillableLimiter. How can I get around this problem? > > *A relevent thread I found on github*: > https://github.com/apache/incubator-gobblin/pull/132 > I understand that the above-mentioned check was added since > the LimitingExtractorDecorator doesn't close the limiters. But why? > > Regards, > Gurjinder >
