Hi Chalitha, This seems to be going through a Hadoop class loader of some kind, which could well not be written in the standard way and would thus be unable to locate the class in question.
Best practice for class loaders is to use the current class's classloader which loading the desired class, but I suspect that Hadoop has its own way of doing this, because it's Hadoop, and because Hadoop is pretty much its own world. If you look at the HDFS connector, it uses another path to get into the file system and doesn't go in through the main door. Thanks, Karl On Tue, Mar 8, 2016 at 12:13 AM, Chalitha Perera <[email protected]> wrote: > Hi All, > > We are writing a new transformation connector[1] to utilize MICO > platform[2]. Platform back-end uses hadoop for storage. > When trying to get the outputstream within connector which is a > HdfsDataOutputStream I am getting class not found exception with > DistributedFileSystem. ManifoldCF already includes hadoop-hdfs jar in > connectors-lib which should contain this class. > > This error is generated only when used within manifoldcf. We have tried the > platform utilization code from a separate jar and it worked without any > errors. > > Any help for this matter is greatly appreciated. > > Here is the error I am getting, > > Error tossed: java.lang.ClassNotFoundException: Class > org.apache.hadoop.hdfs.DistributedFileSystem not found > java.lang.RuntimeException: java.lang.ClassNotFoundException: Class > org.apache.hadoop.hdfs.DistributedFileSystem not found > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2074) > at > org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2578) > at > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591) > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) > at > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169) > at > > eu.mico.platform.storage.impl.StorageServiceHDFS.getOutputStream(StorageServiceHDFS.java:62) > at > > eu.mico.platform.persistence.impl.MarmottaContent.getOutputStream(MarmottaContent.java:266) > at > > org.apache.manifoldcf.agents.transformation.mico.MicoExtractor.addOrReplaceDocumentWithException(MicoExtractor.java:182) > at > > org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddEntryPoint.addOrReplaceDocumentWithException(IncrementalIngester.java:3221) > at > > org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineAddFanout.sendDocument(IncrementalIngester.java:3072) > at > > org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester$PipelineObjectWithVersions.addOrReplaceDocumentWithException(IncrementalIngester.java:2706) > at > > org.apache.manifoldcf.agents.incrementalingest.IncrementalIngester.documentIngest(IncrementalIngester.java:756) > at > > org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1583) > at > > org.apache.manifoldcf.crawler.system.WorkerThread$ProcessActivity.ingestDocumentWithException(WorkerThread.java:1548) > at > > org.apache.manifoldcf.crawler.connectors.filesystem.FileConnector.processDocuments(FileConnector.java:404) > at > > org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:399) > Caused by: java.lang.ClassNotFoundException: Class > org.apache.hadoop.hdfs.DistributedFileSystem not found > at > > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1980) > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2072) > > > Thanks, > Chalitha > > > [1] https://github.com/zaizi/sensefy-mico-connector > [2] https://bitbucket.org/mico-project/platform > > -- > > ------------------------------ > This message should be regarded as confidential. If you have received this > email in error please notify the sender and destroy it immediately. > Statements of intent shall only become binding when confirmed in hard copy > by an authorised signatory. > > Zaizi Ltd is registered in England and Wales with the registration number > 6440931. The Registered Office is Brook House, 229 Shepherds Bush Road, > London W6 7AN. >
