[
https://issues.apache.org/jira/browse/SOLR-14660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17167449#comment-17167449
]
Istvan Farkas commented on SOLR-14660:
--------------------------------------
I'm working on the HDFS package. As discussed with [~ichattopadhyaya], the
first step of the plan is to move the HDFS code to a module that builds as a
separate jar file and remove the direct HDFS references from the core.
Loading the DirectoryFactory from a separate jar should already be possible.
Loading it from a package requires the jira [~noble.paul] is working on.
I found the following reference points to hdfs:
- CheckHdfsIndex - standalone utility
- HdfsBackupRepository - used from solr.xml
- HdfsDirectoryFactory (from solrconfig.xml), also referenced by the
HdfsBackupRepository, CheckHdfsIndex
- HdfsUpdateLog (invoked from UpdateHandler, // if (dirFactory instanceof
HdfsDirectoryFactory)
- BlockDirectory also references hdfs directory
So far I moved the HDFS-related code and tests to a contrib module and created
gradle build for it and also removed the references from core to HDFS.
The BlockDirectory reference was an instanceof check:
{code}
private long getFileModified(String name) throws IOException {
if (in instanceof FSDirectory) {
File directory = ((FSDirectory) in).getDirectory().toFile();
File file = new File(directory, name);
if (!file.exists()) {
throw new FileNotFoundException("File [" + name + "] not found");
}
return file.lastModified();
} else if (in instanceof HdfsDirectory) {
return ((HdfsDirectory) in).fileModified(name);
} else {
throw new UnsupportedOperationException();
}
}
{code}
I added the getLastModified to the base class and the implementations to
FSDirectory and HdfsDirectory.
Also the updatehandler had some direct references to the HdfsDirectory, I
removed those.
At the unit test level, the tests which are supposed to test Hadoop-level
operations like hadoop authentication are also referencing the HDFS code, it
will need some additional work to clean up those.
I will work on stabilizing this as a separate jar and then sync up with
[~ichattopadhyaya] and [~noble.paul] on making this a package.
> Migrating HDFS into a package
> -----------------------------
>
> Key: SOLR-14660
> URL: https://issues.apache.org/jira/browse/SOLR-14660
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Ishan Chattopadhyaya
> Priority: Major
>
> Following up on the deprecation of HDFS (SOLR-14021), we need to work on
> isolating it away from Solr core and making a package for this. This issue is
> to track the efforts for that.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]