[
https://issues.apache.org/jira/browse/SOLR-7843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973453#comment-14973453
]
Pablo Lozano commented on SOLR-7843:
------------------------------------
Thanks for fixing it.
I closed it before because I was starting to think it could have been my fault
as I was using the import-handler on a very unorthodox way. Later on I did
realize it was a real issue and it should be fixed but by that time I was using
a different method of delta import that did not trigger this behavior. I forgot
to reopen it again.
This issue only happens when the delta import is huge or use the import-handler
on a very unorthodox way like I did but it should definitely needed to be fixed.
Maybe this discussion should be for other day but even thoe the import-handler
is very good I think it is missing some flexibility to avoid this type of
issues. I think most of the time developers would want to use it as a base and
not as full fledged component. Most of the times use cases are very specific to
business cases and the default implementation looks like something that has
tried to adapt to all possible cases possible. This has constrained the
flexibility on this plugin by adding inflexible edge cases rules, made strange
abstractions and forces a very opinionated workflow.
In my opinion this plugin should serve as a base for developers to implement
their own import functionality and set of tools to help them manage the state
of the import. It is easier for a developer to implement an api than try to
work around a framework.
That is just my two cents of an overall great plugin.
Thanks
> Importing Delta create a memory leak
> ------------------------------------
>
> Key: SOLR-7843
> URL: https://issues.apache.org/jira/browse/SOLR-7843
> Project: Solr
> Issue Type: Bug
> Components: contrib - DataImportHandler
> Affects Versions: 5.2.1
> Reporter: Pablo Lozano
> Assignee: Shalin Shekhar Mangar
> Labels: memory-leak
> Fix For: 5.4, Trunk
>
>
> The org.apache.solr.handler.dataimport.SolrWriter is not correctly cleaning
> itself after finishing importing Deltas as the "Set<Object> deltaKeys" is not
> being cleaned after the process has finished.
> When using a custom importer or DataSource for my case I need to add
> additional parameters to the delta keys.
> When the data import finishes the DeltaKeys is not set back to null and the
> DataImporter, DocBuilder and the SolrWriter are mantained as live objects
> because there are being referenced by the "infoRegistry" of the SolrCore
> which seems to be used for Jmx information.
> It appears that starting a second delta import did not freed the memory which
> may cause on the long run an OutOfMemory, I have not checked if starting a
> full import would break the references and free the memory.
> An easy fix is possible which would be to add to the SolrWriter "deltaKeys =
> null;" on the close method.
> Or nullify the writer on DocBuilder after being used on the method execute();
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
