[
https://issues.apache.org/jira/browse/SOLR-11473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16805482#comment-16805482
]
Kevin Risden commented on SOLR-11473:
-------------------------------------
Attached patch that does the following:
* Uses Hadoop Path for isAbsolute check.
* Adds test to ensure isAbsolute works correctly with non hdfs:// prefixes.
* Ensures that the Path scheme is set to disable the cache when getting from
the filesystem.
The Path approach is just like URI, but Path wraps URI and does some more
Hadoop specific handling. Made sure that we handle the case of
fs.SCHEME.impl.disable.cache=true since that can cause some issues with
filesystem already being closed if that is missed.
> Make HDFSDirectoryFactory support other prefixes (besides hdfs:/)
> -----------------------------------------------------------------
>
> Key: SOLR-11473
> URL: https://issues.apache.org/jira/browse/SOLR-11473
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Hadoop Integration, hdfs
> Affects Versions: 6.6.1
> Reporter: Radu Gheorghe
> Assignee: Kevin Risden
> Priority: Minor
> Fix For: 8.1, master (9.0)
>
> Attachments: SOLR-11473.patch, SOLR-11473.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Not sure if it's a bug or a missing feature :) I'm trying to make Solr work
> on Alluxio, as described by [~thelabdude] in
> https://www.slideshare.net/thelabdude/running-solr-in-the-cloud-at-memory-speed-with-alluxio/1
> The problem I'm facing here is with autoAddReplicas. If I have
> replicationFactor=1 and the node with that replica dies, the node taking over
> incorrectly assigns the data directory. For example:
> before
> {code}"dataDir":"alluxio://localhost:19998/solr/test/",{code}
> after
> {code}"dataDir":"alluxio://localhost:19998/solr/test/core_node1/alluxio://localhost:19998/solr/test/",{code}
> The same happens for ulogDir. Apparently, this has to do with this bit from
> HDFSDirectoryFactory:
> {code} public boolean isAbsolute(String path) {
> return path.startsWith("hdfs:/");
> }{code}
> If I add "alluxio:/" in there, the paths are correct and the index is
> recovered.
> I see a few options here:
> * add "alluxio:/" to the list there
> * add a regular expression in the lines of \[a-z]*:/ I hope that's not too
> expensive, I'm not sure how often this method is called
> * don't do anything and expect alluxio to work with an "hdfs:/" path? I
> actually tried that and didn't manage to make it work
> * have a different DirectoryFactory or something else?
> What do you think?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
