Re: ResourceManager fails to start
Hi Alexandru Can you share whats value in capacity scheduler for yarn.scheduler.capacity.am.failure.scheduling.delay.ms Regards -Sanjeev On Fri, Jun 26, 2015 at 6:40 PM, Alexandru Pacurar alexandru.pacu...@propertyshark.com wrote: Hello, I’m running Hadoop 2.6 and I have encountered a problem with the resourcemanager. After a restart the resourcemanager refuses to start with the following error: 2015-06-26 08:54:10,342 INFO attempt.RMAppAttemptImpl (RMAppAttemptImpl.java:recover(796)) - Recovering attempt: appattempt_1435159945366_0792_01 with final state: null 2015-06-26 08:54:10,342 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:createAndGetAMRMToken(195)) - Create AMRMToken for ApplicationAttempt: appattempt_1435159945366_0792_01 2015-06-26 08:54:10,342 INFO security.AMRMTokenSecretManager (AMRMTokenSecretManager.java:createPassword(307)) - Creating password for appattempt_1435159945366_0792_01 2015-06-26 08:54:10,343 INFO resourcemanager.ApplicationMasterService (ApplicationMasterService.java:registerAppAttempt(670)) - Registering app attempt : appattempt_1435159945366_0792_01 2015-06-26 08:54:10,344 ERROR resourcemanager.ResourceManager (ResourceManager.java:serviceStart(594)) - Failed to load/recover state java.lang.NullPointerException at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.addApplicationAttempt(CapacityScheduler.java:734) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1089) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:114) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1038) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl$AttemptRecoveredTransition.transition(RMAppAttemptImpl.java:1002) at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385) at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:755) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:106) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.recoverAppAttempts(RMAppImpl.java:831) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.access$1900(RMAppImpl.java:101) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:846) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:836) at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385) at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:711) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:312) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:413) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.recover(ResourceManager.java:1207) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:590) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1014) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1051) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1047) at
Grant privileges on / of hdfs file system
Hi, Can anyone provide me steps to grant a particular user access equivalnent to that of hdfs user in hadoop. The reason behind this is I want to have my custom user who can create anything on the entire hdfs file system (/) I tried couple of links however, none of them were useful. Is there any way by adding/modifying some property tags I can do that ? With Regards, Pratik DISCLAIMER == This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: Subscribe
Take a look at http://hadoop.apache.org/mailing_lists.html Cheers On Tue, Jul 7, 2015 at 1:09 AM, Akmal Abbasov akmal.abba...@icloud.com wrote:
Re: get the hadoop working dir using the bash commands?
Hello, Are you looking for the working directory used by HDFS to resolve relative paths used in commands like hdfs dfs -ls myRelativePath? If so, then the working directory is the current user's home directory. HDFS defines the home directory as a common prefix followed by the username. The prefix is controlled by configuration property dfs.user.home.dir.prefix, and the default is /user. property namedfs.user.home.dir.prefix/name value/user/value descriptionThe directory to prepend to user name to get the user's home direcotry. /description /property You mentioned wanting to access this in bash. I think you can combine hdfs getconf with the current user to get it. For example: echo $(hdfs getconf -confKey dfs.user.home.dir.prefix)/$USER /user/chris If you're running a secured cluster with complex auth-to-local name conversion rules, then simply using $USER might not be sufficient. If you find that's the case, then look at using the hadoop kerbname your Kerberos principal command. The hadoop kerbname command alias only exists in trunk right now. For 2.x builds, you can still get the same effect by running the underlying class directly: hadoop org.apache.hadoop.security.HadoopKerberosName your Kerberos principal. I hope this helps. --Chris Nauroth On 7/7/15, 2:33 AM, xeonmailinglist xeonmailingl...@gmail.com wrote: Is it possible to get the hadoop working dir using the bash commands?
Re: Large number of application attempts
Hey, Akmal: Are you running a lots of apps ? Or could you, maybe, share some of file names of those znodes, please ? Thanks Xuan Gong On 7/7/15, 1:23 AM, Akmal Abbasov akmal.abba...@icloud.com wrote: Hi all, I have HBase cluster. I run a cron job each hour to create a snapshot of tables in HBase and export these snapshots to another HBase cluster for backup. But I¹ve noticed that there are ~4000 znodes in /hadoop-ha/testhbase1/rmstore/ZKRMStateRoot/RMAppRoot in Zookeeper. What are these znodes? Are they failed yarn applications? How I need to handle them? Thank you. Akmal
Subscribe
Large number of application attempts
Hi all, I have HBase cluster. I run a cron job each hour to create a snapshot of tables in HBase and export these snapshots to another HBase cluster for backup. But I’ve noticed that there are ~4000 znodes in /hadoop-ha/testhbase1/rmstore/ZKRMStateRoot/RMAppRoot in Zookeeper. What are these znodes? Are they failed yarn applications? How I need to handle them? Thank you. Akmal
Re: Grant privileges on / of hdfs file system
Hello Pratik, The hdfs user (more specifically, whatever user launched the NameNode process) is the HDFS super-user. The super-user has full access to the file system and also administrative operations. You can declare additional users to be super-users by setting property dfs.permissions.superusergroup in hdfs-site.xml. The default value of this property is supergroup. property namedfs.permissions.superusergroup/name valuesupergroup/value descriptionThe name of the group of super-users./description /property Any user you add to group supergroup (or whatever custom group you use if you decided to change dfs.permissions.superusergroup) will be treated as an HDFS super-user. It's important to keep in mind that this grants both full file system access and full administrative access. That means the user would be able to call sensitive operations like hdfs dfsadmin -safemode enter. If this isn't appropriate, then you might explore using file system permissions and ACLs to implement your requirements on the file system only. More details are in the documentation here: http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html I hope this helps. --Chris Nauroth From: Pratik Gadiya pratik_gad...@persistent.commailto:pratik_gad...@persistent.com Reply-To: user@hadoop.apache.orgmailto:user@hadoop.apache.org user@hadoop.apache.orgmailto:user@hadoop.apache.org Date: Tuesday, July 7, 2015 at 9:38 AM To: user@hadoop.apache.orgmailto:user@hadoop.apache.org user@hadoop.apache.orgmailto:user@hadoop.apache.org Subject: Grant privileges on / of hdfs file system Hi, Can anyone provide me steps to grant a particular user access equivalnent to that of hdfs user in hadoop. The reason behind this is I want to have my custom user who can create anything on the entire hdfs file system (/) I tried couple of links however, none of them were useful. Is there any way by adding/modifying some property tags I can do that ? With Regards, Pratik DISCLAIMER == This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: Problem when configure the security in hadoop
?Not sure about HDFS special setup, but in general, to use HTTPs, you should have your keystore/truststore generated and config ssl-client.xml and ssl-server.xml properly. - Zhijie From: Colin Ma colin...@gmail.com Sent: Friday, July 03, 2015 1:37 AM To: user@hadoop.apache.org Subject: Problem when configure the security in hadoop Hi, I do the security configuration for Hadoop these days, the Kerberos works fine, but there maybe has some problems on sasl configuration. The following is the related configuration in hdfs-site.xml: property namedfs.http.policy/name valueHTTPS_ONLY/value /property property namedfs.data.transfer.protection/name valueauthentication/value /property There is no problem to execute the command like: hdfs dfs -ls / But when I execute the command: hdfs dfs -copyToLocal /temp/test.txt . The following exception will be thrown: 015-07-03 14:02:54,715 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Added bpid=BP-271423801-192.168.20.28-1423724265164 to blockPoolScannerMap, new size=1 2015-07-03 14:03:39,963 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: server-511:50010:DataXceiver error processing unknown operation src: /192.168.20.28:58422http://192.168.20.28:58422 dst: /192.168.20.28:50010http://192.168.20.28:50010 java.io.EOFException: Premature EOF: no length prefix available at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2203) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil.readSaslMessageAndNegotiationCipherOptions(DataTransferSaslUtil.java:233) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.doSaslHandshake(SaslDataTransferServer.java:369) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.getSaslStreams(SaslDataTransferServer.java:297) at org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferServer.receive(SaslDataTransferServer.java:124) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:183) at java.lang.Thread.run(Thread.java:745) 2015-07-03 15:34:39,917 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Sent 1 blockreports 145 blocks total. Took 1 msec to generate and 6 msecs for RPC and NN processing. Got back commands org.apache.hadoop.hdfs.server.protocol.FinalizeCommand@1b3bce82mailto:org.apache.hadoop.hdfs.server.protocol.FinalizeCommand@1b3bce82 Just take a look the method doSaslHandshake() of SaslDataTransferClient.java and SaslDataTransferServer.java, maybe SaslDataTransferClient send a empty response cause this exception, and I think some mistakes in the configuration caused this problem. Is there anyone can help to check this problem? Thanks for your help. Best regards, Colin Ma
return tuple of key value pair from string in java UDF
Hi, I have a pig script which return field as Key=ValKey=ValKey=ValKey=ValKey=Val whose schema is defined as data:chararray in my pig script Can I use java UDF to return a tuple of Key Value Pair. Would really appreciate if somebody help me point to some guide or example of this kind of scenario. Thanks
unsubscribe
unsubscribe Best regards Will. William J Bell Sales Director – www.wandisco.com +44 7825 770477 - will.b...@wandisco.com Believed technically impossible, WANdisco customers now exploit 100% uptime and exact real-time replication across widely distributed Hadoop, Subversion and Git. On 7 Jul 2015, at 10:44, Divya Gehlot divya.htco...@gmail.com wrote: Hi, I have a pig script which return field as Key=ValKey=ValKey=ValKey=ValKey=Val whose schema is defined as data:chararray in my pig script Can I use java UDF to return a tuple of Key Value Pair. Would really appreciate if somebody help me point to some guide or example of this kind of scenario. Thanks -- Learn how WANdisco Fusion solves Hadoop data protection and scalability challenges http://www.wandisco.com/hadoop/wd-fusion Listed on the London Stock Exchange: WAND http://www.bloomberg.com/quote/WAND:LN THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its subsidiaries, (WANdisco) does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of WANdisco, unless the author is authorized by WANdisco to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by WANdisco. Although WANdisco operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed.
get the hadoop working dir using the bash commands?
Is it possible to get the hadoop working dir using the bash commands?
