isonusingh opened a new issue, #11236:
URL: https://github.com/apache/pinot/issues/11236
Hi Experts,
I am trying to setup HDFS as deep storage for Apache Pinot.
Below is the error I am facing:
Did not find any fs classes in the configuration
Got scheme hdfs, initializing class
org.apache.pinot.plugin.filesystem.HadoopPinotFS
Initializing PinotFS for scheme hdfs, classname
org.apache.pinot.plugin.filesystem.HadoopPinotFS
Unable to load native-hadoop library for your platform... using builtin-java
classes where applicable
Could not instantiate file system for class
org.apache.pinot.plugin.filesystem.HadoopPinotFS with scheme hdfs
java.lang.RuntimeException: Could not initialize HadoopPinotFS
at
org.apache.pinot.plugin.filesystem.HadoopPinotFS.init(HadoopPinotFS.java:71)
Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No
FileSystem for scheme "hdfs"
java.lang.RuntimeException: java.lang.RuntimeException: Could not initialize
HadoopPinotFS
Below configuration I added in conf file:
controller.data.dir=hdfs://master:8020/apps/pinot/controller/segment
controller.local.temp.dir=/tmp/pinot/
controller.zk.str=<demo-edge:2191>
controller.enable.split.commit=true
controller.access.protocols.http.port=9000
controller.helix.cluster.name=PinotCluster
pinot.controller.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS
pinot.controller.storage.factory.hdfs.hadoop.conf.path=/usr/hdp/hadoop/etc/hadoop/
pinot.controller.segment.fetcher.protocols=file,http,hdfs
pinot.controller.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
controller.vip.port=9000
controller.port=9000
pinot.set.instance.id.to.hostname=true
pinot.server.grpc.enable=true
-------------------------
pinot.server.instance.enable.split.commit=true
pinot.server.storage.factory.class.hdfs=org.apache.pinot.plugin.filesystem.HadoopPinotFS
pinot.server.storage.factory.hdfs.hadoop.conf.path=/usr/hdp/hadoop/etc/hadoop/
pinot.server.segment.fetcher.protocols=file,http,hdfs
pinot.server.segment.fetcher.hdfs.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.set.instance.id.to.hostname=true
pinot.server.instance.dataDir=/opt/pinot/data/server/index
pinot.server.instance.segmentTarDir=/opt/pinot/data/server/segment
pinot.server.grpc.enable=true
pinot.server.grpc.port=8090
And Already exported below Hadoop details:
export HADOOP_HOME=/usr/hdp/hadoop
export HADOOP_VERSION=3.1.0
export GC_LOG_LOCATION=/opt/pinot/logs
export PINOT_VERSION=0.7.1
export PINOT_DISTRIBUTION_DIR=/opt/pinot/
export SERVER_CONF_DIR=/opt/pinot/conf/
export ZOOKEEPER_ADDRESS=localhost:2191
export HADOOP_GUAVA_VERSION=15.0
export HADOOP_GSON_VERSION=2.2.4
export
CLASSPATH_PREFIX="${HADOOP_HOME}/share/hadoop/hdfs/hadoop-hdfs-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-annotations-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/hadoop-auth-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/hadoop-common-${HADOOP_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/guava-${HADOOP_GUAVA_VERSION}.jar:${HADOOP_HOME}/share/hadoop/common/lib/gson-${HADOOP_GSON_VERSION}.jar:${HADOOP_HOME}/share/hadoop/hdfs/lib/commons-configuration2-2.1.1.jar:${HADOOP_HOME}/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
