Hi,
I have a hadoop cluster installed using HDP3.0 and ambari 2.7 blueprint. I
have copied oracle oci hdfs connector jar (downloaded from
https://github.com/oracle/oci-hdfs-connector/releases) to
/usr/hdp/3.0.0.0-1634/livy2/repl_2.11-jars/
on namenodes and datanodes by following
https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.0.0/using-zeppelin/zeppelin-using.pdf
.
When using livy2.spark interpreter in Zeppelin 0.8.0 (which use yarn
cluster mode), I got mixed story.
This works which means the oci connector hdfs jar is imported successfully.
%livy2.spark
import com.oracle.bmc.hdfs._
This failed:
%livy2.spark
val df = spark.read.parquet("oci://mybucket@mytenancy/my.parquet")
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for
scheme "oci" at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3332) at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3352) at
org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3403) at
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3371) at
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:477) at
org.apache.hadoop.fs.Path.getFileSystem(Path.java:361) at
org.apache.spark.sql.execution.datasources.DataSource$.org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary(DataSource.scala:705)
at
org.apache.spark.sql.execution.datasources.DataSource$$anonfun$15.apply(DataSource.scala:389)
at
org.apache.spark.sql.execution.datasources.DataSource$$anonfun$15.apply(DataSource.scala:389)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at scala.collection.immutable.List.foreach(List.scala:381) at
scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:241)
at scala.collection.immutable.List.flatMap(List.scala:344) at
org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:388)
at
org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:239)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:227) at
org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:622) at
org.apache.spark.sql.DataFrameReader.parquet(DataFrameReader.scala:606) ...
60 elided
Any idea? Appreciate very much!
