I have seen this issue before in OpenStack environment where the DFS folders were configured to /tmp/… and they were wiped out after restart.
If it is a test env, you can try formatting. First check dfs.name.dir property in HDFS. % $HADOOP_INSTALL/hadoop/bin/hadoop namenode -format http://wiki.apache.org/hadoop/GettingStartedWithHadoop Formatting the Namenode The first step to starting up your Hadoop installation is formatting the Hadoop filesystem, which is implemented on top of the local filesystems of your cluster. You need to do this the first time you set up a Hadoop installation. Do not format a running Hadoop filesystem, this will cause all your data to be erased. Before formatting, ensure that the dfs.name.dir directory exists. If you just used the default, then mkdir -p /tmp/hadoop-username/dfs/name will create the directory. To format the filesystem (which simply initializes the directory specified by the dfs.name.dir variable), run the command: % $HADOOP_INSTALL/hadoop/bin/hadoop namenode -format If asked to [re]format, you must reply Y (not just y) if you want to reformat, else Hadoop will abort the format. Thanks Bosco From: Aneela Saleem <ane...@platalytics.com> Reply-To: "user@ranger.incubator.apache.org" <user@ranger.incubator.apache.org> Date: Tuesday, July 28, 2015 at 12:40 PM To: "user@ranger.incubator.apache.org" <user@ranger.incubator.apache.org> Subject: Re: Error Setting up Ranger-Admin > NameNode error Logs: > > 2015-06-30 00:30:05,933 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: > Failed to start namenode. > java.io.IOException: NameNode is not formatted. > > On Wed, Jul 29, 2015 at 12:22 AM, Don Bosco Durai <bo...@apache.org> wrote: >> Anything in the namenode log file? >> >> Thanks >> >> Bosco >> >> From: Aneela Saleem <ane...@platalytics.com> >> Reply-To: "user@ranger.incubator.apache.org" >> <user@ranger.incubator.apache.org> >> Date: Tuesday, July 28, 2015 at 12:18 PM >> To: "user@ranger.incubator.apache.org" <user@ranger.incubator.apache.org> >> Subject: Re: Error Setting up Ranger-Admin >> >>> Hdfs Plugin request is not being shown in Audit > Plugins tab. I followed >>> all steps and verified with your steps. Restarted namenode but still can't >>> see any plugin request >>> >>> On Wed, Jul 29, 2015 at 12:03 AM, Aneela Saleem <ane...@platalytics.com> >>> wrote: >>>> Thank you so much Bosco. I created JIRA for this. Can be found here >>>> <https://issues.apache.org/jira/browse/RANGER-582> >>>> >>>> Let me try UserSync, then i will approach yyou in case of any problem. >>>> >>>> On Tue, Jul 28, 2015 at 11:28 PM, Don Bosco Durai <bdu...@hortonworks.com> >>>> wrote: >>>>> Good. I already typed it, so let me paste it for the benefit of others. We >>>>> need to update the doc also. If you don’t mind, can you create a JIRA to >>>>> track this? >>>>> 1. Go to RangerAdmin UI -> AccessManager (first menu tab is already opened >>>>> by default) >>>>> 2. Click on “+” in the HDFS panel title >>>>> 3. Enter Service Name: e.g. “${cluster_name}_hdfs. Cluster name could be >>>>> anything. You can call the repo “test_hdfs” if you want to >>>>> 4. UserName: ranger (doesn’t matter whether it exists because you are not >>>>> using kerberos) >>>>> 5. Password: ranger (doesn’t matter what you give) >>>>> 6. Namenode URL: hdfs://<namenode_host>:8020 >>>>> 7. Authorization Enabled: No >>>>> 8. Authentication Type: Simple >>>>> 9. Rest you can leave it as default. >>>>> Save this screen. Then use the same name e.g “test_hdfs” in your >>>>> install.properties for enabling Ranger in HDFS. Then restart namenode. >>>>> >>>>> Is the plugin request showing up in the Audit->Plugin tab? >>>>> >>>>> If the plugin is showing up, then that means authorization and auditing is >>>>> already working. >>>>> >>>>> For configuring policies, we need to configure UserSync. The >>>>> install/configure process is same for all processes. You need to update >>>>> the install.properties with the appropriate values. >>>>> Set SYNC_SOURCE = ldap >>>>> And check the section "Table: LDAP/AD Properties with sample values” for >>>>> reference values. >>>>> >>>>> Please note that for using LDAP within Hadoop, you need to configure >>>>> Hadoop core-site.xml to use LDAP. You might be already knowing it. Just >>>>> want to make sure. >>>>> >>>>> Thanks >>>>> >>>>> Bosco >>>>> >>>>> >>>>> From: Aneela Saleem <ane...@platalytics.com> >>>>> Reply-To: "user@ranger.incubator.apache.org" >>>>> <user@ranger.incubator.apache.org> >>>>> Date: Tuesday, July 28, 2015 at 11:13 AM >>>>> >>>>> To: "user@ranger.incubator.apache.org" <user@ranger.incubator.apache.org> >>>>> Subject: Re: Error Setting up Ranger-Admin >>>>> >>>>>> Luckily i have created the service myself. Thanks for your timely help. >>>>>> >>>>>> Can you please tell me now what should be the next step. I want to >>>>>> explore all the functionality i.e., >>>>>> >>>>>> sync LDAP contacts >>>>>> perform authorization etc >>>>>> >>>>>> On Tue, Jul 28, 2015 at 11:07 PM, Aneela Saleem <ane...@platalytics.com> >>>>>> wrote: >>>>>>> There are no errors in NameNode logs. >>>>>>> >>>>>>> Can you please tell me how to create service/repo using Ranger Admin? >>>>>>> >>>>>>> On Tue, Jul 28, 2015 at 10:48 PM, Don Bosco Durai <bo...@apache.org> >>>>>>> wrote: >>>>>>> Yes, this is link I was about to redirect you to. >>>>>>> >>>>>>> Sorry, I might have assumed incorrectly on your initial question. >>>>>>> ./setup.sh is called only for installing RangerAdmin. Good it seems you >>>>>>> have made progress on installing RangerAdmin. >>>>>>> >>>>>>> I also assume, your ./enable-hdfs-plugin.sh went through fine. >>>>>>> >>>>>>> Looking into the Apache Ranger document, it seems it doesn’t explicitly >>>>>>> call out to create a repo/service for HDFS in RangerAdmin. In the >>>>>>> install.properties for HDFS, what value have you given for the property >>>>>>> “REPOSITORY_NAME”? You need to create a service/repo in RangerAdmin with >>>>>>> the same name. >>>>>>> >>>>>>> Also, after you restart HDFS namenode, can you check the namenode logs >>>>>>> for any error? >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Bosco >>>>>>> >>>>>>> >>>>>>> From: Aneela Saleem <ane...@platalytics.com> >>>>>>> Reply-To: "user@ranger.incubator.apache.org" >>>>>>> <user@ranger.incubator.apache.org> >>>>>>> Date: Tuesday, July 28, 2015 at 10:16 AM >>>>>>> >>>>>>> To: "user@ranger.incubator.apache.org" >>>>>>> <user@ranger.incubator.apache.org> >>>>>>> Subject: Re: Error Setting up Ranger-Admin >>>>>>> >>>>>>> And i'm using following link for installation: >>>>>>> >>>>>>> https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+0.5.0+I >>>>>>> nstallation >>>>>>> >>>>>>> I'm upto Enabling Ranger HDFS Plugins section (e) >>>>>>> * You can verify by logging into the Ranger Admin Web interface > Audit >>>>>>> > Agents. >>>>>>> * You can verify the plugin is communicating to Ranger admin in >>>>>>> Audit->plugins tab >>>>>>> >>>>>>> >>>>>>> where i don't see any audits ensuring hdfs plugin is communicating to >>>>>>> Ranger Admin. >>>>>>> >>>>>>> On Tue, Jul 28, 2015 at 10:11 PM, Aneela Saleem <ane...@platalytics.com> >>>>>>> wrote: >>>>>>> I'm planning to secure HDFS and i'm using Knox for authentication using >>>>>>> REST API. >>>>>>> >>>>>>> Yes i'm still able to access the RangerAdmin UI. I'm just curious how to >>>>>>> sync LDAP contacts in APache Ranger. >>>>>>> >>>>>>> Secondly if the document is related to 0.4 and i'm linked with 0.5, then >>>>>>> how should i proceed? Do you have any fruitful link, i should follow? >>>>>>> (being a novice) >>>>>>> >>>>>>> On Tue, Jul 28, 2015 at 10:04 PM, Don Bosco Durai <bo...@apache.org> >>>>>>> wrote: >>>>>>> Aneela >>>>>>> >>>>>>> The document you linked is for Apache Ranger 0.4. There subtle changes >>>>>>> in Apache Ranger 0.5. >>>>>>> >>>>>>> Just curious, how you were able to access RangerAdmin UI? Did the >>>>>>> installation continued after the install failure? >>>>>>> >>>>>>> Also, can you give additional information of your env? I can try >>>>>>> reproducing it. >>>>>>> >>>>>>> Also, which components are you planning secure? (HDFS, Hive, Hbase, >>>>>>> Solr, YARN, Storm, etc) >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Bosco >>>>>>> >>>>>>> >>>>>>> From: Aneela Saleem <ane...@platalytics.com> >>>>>>> Reply-To: "user@ranger.incubator.apache.org" >>>>>>> <user@ranger.incubator.apache.org> >>>>>>> Date: Tuesday, July 28, 2015 at 5:19 AM >>>>>>> To: "user@ranger.incubator.apache.org" >>>>>>> <user@ranger.incubator.apache.org> >>>>>>> Subject: Re: Error Setting up Ranger-Admin >>>>>>> >>>>>>> And secondly my Ranger UI does not Match the UI as shown in the >>>>>>> following link: >>>>>>> >>>>>>> http://pivotalhd.docs.pivotal.io/docs/ranger-user-guide.html#Item1.4.2 >>>>>>> >>>>>>> There is no Policy Manager Tab and no Manage Repository Thing. >>>>>>> >>>>>>> On Tue, Jul 28, 2015 at 5:17 PM, Aneela Saleem <ane...@platalytics.com> >>>>>>> wrote: >>>>>>> Hi Bosco, >>>>>>> >>>>>>> I did as you suggested but still getting same error. >>>>>>> >>>>>>> On Tue, Jul 28, 2015 at 4:49 AM, Don Bosco Durai <bo...@apache.org> >>>>>>> wrote: >>>>>>> The properties file looks good.. >>>>>>> >>>>>>> I am not sure whether it is because of python 2.7. Is it possible for >>>>>>> you to use python 2.6? >>>>>>> >>>>>>> Try changing the below to where 2.6 is installed. >>>>>>> >>>>>>> PYTHON_COMMAND_INVOKER=python >>>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> Bosco >>>>>>> >>>>>>> >>>>>>> From: Aneela Saleem <ane...@platalytics.com> >>>>>>> Reply-To: "user@ranger.incubator.apache.org" >>>>>>> <user@ranger.incubator.apache.org> >>>>>>> Date: Monday, July 27, 2015 at 2:02 PM >>>>>>> To: "user@ranger.incubator.apache.org" >>>>>>> <user@ranger.incubator.apache.org> >>>>>>> Subject: Error Setting up Ranger-Admin >>>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I have followed this link >>>>>>> >>>>>>> https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+0.5.0+I >>>>>>> nstallation >>>>>>> >>>>>>> But when i run ./setupsh, i get the following error at the end: >>>>>>> >>>>>>> Traceback (most recent call last): >>>>>>> File "update_property.py", line 40, in <module> >>>>>>> >>>>>>> write_properties_to_xml(ranger_admin_site_xml_path,parameter_name,parame >>>>>>> ter_value) >>>>>>> File "update_property.py", line 21, in write_properties_to_xml >>>>>>> if(os.path.isfile(xml_path)): >>>>>>> File "/usr/lib/python2.7/genericpath.py", line 29, in isfile >>>>>>> st = os.stat(path) >>>>>>> TypeError: coercing to Unicode: need string or buffer, NoneType found >>>>>>> 2015-07-28 01:38:57,308 [E] Update property failed for: >>>>>>> >>>>>>> Attached is the install.proprties file. >>>>>>> >>>>>>> I'm new to Apache Knox, can anyone please guide me. I'm having troubles >>>>>>> in installing Apache Ranger. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>> >>> >
