Hi Bala, Are you running a single-machine HBase setup? The ZooKeeper embedded in such a setup is pretty fragile to disk space issue and your ZNode might have corrupted.
If that’s indeed your setup, please take a look at HBase log files, specifically on messages from ZooKeeper. In this situation, one way to recover is to remove ZooKeeper files and let HBase recreate them, assuming from your log output that you don’t have other services depend on the same ZK. Regards, Donald On Thu, Apr 12, 2018 at 5:34 AM bala vivek <bala.vivek...@gmail.com> wrote: > Hi, > > I use PIO 0.10.0 version and hbase 1.2.4. The setup was working fine till > today morning. I saw PIO was down as the mount space issue was present on > the server and cleared the unwanted files. > > After doing a pio-stop-all and pio-start-all the HMaster service is not > working. I tried multiple times the pio restart. > > I can see whenever I do a pio-stop-all and check the service using jps, > the Hmaster seems running. Similarly I tried to run the ./start-hbase.sh > script but still pio status is not showing as success. > > pio error log : > > [INFO] [Console$] Inspecting PredictionIO... > [INFO] [Console$] PredictionIO 0.10.0-incubating is installed at > /opt/tools/PredictionIO-0.10.0-incubating > [INFO] [Console$] Inspecting Apache Spark... > [INFO] [Console$] Apache Spark is installed at > /opt/tools/PredictionIO-0.10.0-incubating/vendors/spark-1.6.3-bin-hadoop2.6 > [INFO] [Console$] Apache Spark 1.6.3 detected (meets minimum requirement > of 1.3.0) > [INFO] [Console$] Inspecting storage backend connections... > [INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)... > [INFO] [Storage$] Verifying Model Data Backend (Source: LOCALFS)... > [INFO] [Storage$] Verifying Event Data Backend (Source: HBASE)... > [ERROR] [RecoverableZooKeeper] ZooKeeper exists failed after 1 attempts > [ERROR] [ZooKeeperWatcher] hconnection-0x7c891ba7, quorum=localhost:2181, > baseZNode=/hbase Received unexpected KeeperException, re-throwing exception > [WARN] [ZooKeeperRegistry] Can't retrieve clusterId from Zookeeper > [ERROR] [StorageClient] Cannot connect to ZooKeeper (ZooKeeper ensemble: > localhost). Please make sure that the configuration is pointing at the > correct ZooKeeper ensemble. By default, HBase manages its own ZooKeeper, so > if you have not configured HBase to use an external ZooKeeper, that means > your HBase is not started or configured properly. > [ERROR] [Storage$] Error initializing storage client for source HBASE > [ERROR] [Console$] Unable to connect to all storage backends successfully. > The following shows the error message from the storage backend. > [ERROR] [Console$] Data source HBASE was not properly initialized. > (org.apache.predictionio.data.storage.StorageClientException) > [ERROR] [Console$] Dumping configuration of initialized storage backend > sources. Please make sure they are correct. > [ERROR] [Console$] Source Name: ELASTICSEARCH; Type: elasticsearch; > Configuration: TYPE -> elasticsearch, HOME -> > /opt/tools/PredictionIO-0.10.0-incubating/vendors/elasticsearch-1.7.3 > [ERROR] [Console$] Source Name: LOCALFS; Type: localfs; Configuration: > PATH -> /root/.pio_store/models, TYPE -> localfs > [ERROR] [Console$] Source Name: HBASE; Type: (error); Configuration: > (error) > > > Regards, > Bala >