[
https://issues.apache.org/jira/browse/SOLR-11335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16158868#comment-16158868
]
Timothy Potter commented on SOLR-11335:
---------------------------------------
With a bit more digging, I see that HdfsUpdateLog sets
{{fs.hdfs.impl.disable.cache=true}}, but I'm not sure if that Configuration
applies to HdfsDirectory as well. Then I realized that the initialization of
HdfsUpdateLog depends on this check:
{code}
if (dataDir != null && dataDir.startsWith("hdfs:/")) {
{code}
Which doesn't work for me, because my HDFS path is {{alluxio://}}. Given HDFS
allows different schemes besides HDFS, seems like this code should not only
look for hdfs:/
> HdfsDirectory & Factory should not close the FileSystem object retrieved with
> get
> ---------------------------------------------------------------------------------
>
> Key: SOLR-11335
> URL: https://issues.apache.org/jira/browse/SOLR-11335
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Hadoop Integration
> Reporter: Timothy Potter
>
> I'm seeing issues where the Hadoop FileSystem instance is closed out from
> under other objects. From what I understand, the Hadoop FileSystem object
> (org.apache.hadoop.fs.FileSystem) retrieved with {{FileSystem.get}} as is
> done in HdfsDirectory's ctor is a shared object that if closed, can affect
> other code using that same shared instance. You can see this is a cached,
> shared object here ->
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L455
> Thus, I suspect Solr should not be closing any FileSystem instance retrieved
> with get. It's important to mention that if I set the
> {{fs.$SCHEME.impl.disable.cache}} to true, then my problems go away, which
> seems to confirm that Solr is using the API incorrectly. That being said, I'm
> surprised this hasn't been raised before, so maybe I've missed something
> basic in Solr's use of HDFS?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
