hadoop configure effect without restart
Hi, https://issues.apache.org/jira/browse/HADOOP-7001 I also have this kind of need, my environment is hadoop2.3cdh5.0.2, This feature is included in this version hadoop? Or I need patch this? How to use this feature? Is there a demo? thanks Best Regards, Evan
fs.s3a.endpoint not working
Hi All, New to this list. Looking for a bit of help: I'm having trouble connecting Hadoop to a S3-compatable (non AWS) object store. This issue was discussed, but left unresolved, in this thread: https://mail-archives.apache.org/mod_mbox/spark-user/201507.mbox/%3cca+0w_au5es_flugzmgwkkga3jya1asi3u+isjcuymfntvnk...@mail.gmail.com%3E And here, on Cloudera's forums (the second post is mine): https://community.cloudera.com/t5/Data-Ingestion-Integration/fs-s3a-endpoint-ignored-in-hdfs-site-xml/m-p/33694#M1180 I'm running Hadoop 2.6.3 with Java 1.8 (65) on a Linux host. Using Hadoop, I'm able to connect to S3 on AWS, and e.g., list/put/get files. However, when I point the fs.s3a.endpoint configuration directive at my non-AWS S3-Compatable object storage, it appears to still point at (and authenticate against) AWS. I've checked and double-checked my credentials and configuration using both Python's boto library and the s3cmd tool, both of which connect to this non-AWS data store just fine. Any help would be much appreciated. Thanks! -- Caleb Phillips, Ph.D. Data Scientist | Computational Science Center National Renewable Energy Laboratory (NREL) 15013 Denver West Parkway | Golden, CO 80401 303-275-4297 | caleb.phill...@nrel.gov - To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional commands, e-mail: user-h...@hadoop.apache.org
Re: diagnosing the difference between dfs 'du' and 'df'
Just a guess, but could you please check what is your dfs.replication set to ? You should be able to find that setting in hdfs-site.xml or in core-site.xml Thanks Anu On 12/21/15, 6:21 PM, "Martin Serrano" wrote: >Hi, > >I have an application that is writing data rapidly directly to HDFS >(creates and appends) as well as to HBase (10-15 tables). The disk free >for the filesystem will report that a large percentage of the system is >in use: > >$ hdfs dfs -df -h / >Filesystem Size Used Available Use% >hdfs://ha 882.6 G 472.6 G409.9 G 54% > >Yet when I try to figure out where the disk space is being used using >dfs -du reports: > >$ hdfs dfs -du -h / >0/app-logs >7.6 G/apps >382.2 M /hdp >0/mapred >0/mr-history >8.5 K/tmp >3.8 G/user > >A dfsadmin -report during the same time frame is below. I'm trying to >figure out where all of this space is going to. When my application is >killed or quiescent, the df and dfsadmin reports fall in line with what >I would expect. I'm running HDP 2.3 with a default configuration as set >up by Ambari. I'm looking for hints or suggestions on how I can >investigate this issue. It seems crazy that ingesting 12g or so of data >can temporarily consume (reserve?) ~300g of HDFS. > >Thanks, >Martin > >Configured Capacity: 947644268544 (882.56 GB) >Present Capacity: 947064596261 (882.02 GB) >DFS Remaining: 490046627240 (456.39 GB) >DFS Used: 457017969021 (425.63 GB) >DFS Used%: 48.26% >Under replicated blocks: 0 >Blocks with corrupt replicas: 0 >Missing blocks: 0 >Missing blocks (with replication factor 1): 0 > >- >Live datanodes (3): > >Name: *.*.*.*:50010 (**.com) >Hostname: **.com >Decommission Status : Normal >Configured Capacity: 315881422848 (294.19 GB) >DFS Used: 218955099179 (203.92 GB) >Non DFS Used: 168255175 (160.46 MB) >DFS Remaining: 96758068494 (90.11 GB) >DFS Used%: 69.32% >DFS Remaining%: 30.63% >Configured Cache Capacity: 0 (0 B) >Cache Used: 0 (0 B) >Cache Remaining: 0 (0 B) >Cache Used%: 100.00% >Cache Remaining%: 0.00% >Xceivers: 15 >Last contact: Mon Dec 21 17:17:38 EST 2015 > > >Name: *.*.*.*:50010 (**.com) >Hostname: **.com >Decommission Status : Normal >Configured Capacity: 315881422848 (294.19 GB) >DFS Used: 218873337575 (203.84 GB) >Non DFS Used: 151608508 (144.59 MB) >DFS Remaining: 96856476765 (90.20 GB) >DFS Used%: 69.29% >DFS Remaining%: 30.66% >Configured Cache Capacity: 0 (0 B) >Cache Used: 0 (0 B) >Cache Remaining: 0 (0 B) >Cache Used%: 100.00% >Cache Remaining%: 0.00% >Xceivers: 16 >Last contact: Mon Dec 21 17:17:38 EST 2015 > > >Name: *.*.*.*:50010 (*.com) >Hostname: ***.com >Decommission Status : Normal >Configured Capacity: 315881422848 (294.19 GB) >DFS Used: 19189532267 (17.87 GB) >Non DFS Used: 259808600 (247.77 MB) >DFS Remaining: 296432081981 (276.07 GB) >DFS Used%: 6.07% >DFS Remaining%: 93.84% >Configured Cache Capacity: 0 (0 B) >Cache Used: 0 (0 B) >Cache Remaining: 0 (0 B) >Cache Used%: 100.00% >Cache Remaining%: 0.00% >Xceivers: 16 >Last contact: Mon Dec 21 17:17:39 EST 2015 > > > >- >To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org >For additional commands, e-mail: user-h...@hadoop.apache.org > >
RE: [Classpath Issue]NoClassFoundException occurs when depending on the 3rd jar
Make sure you call job.setJarByClass with right parameters. http://stackoverflow.com/questions/3912267/hadoop-query-regarding-setjarbyclass-method-of-job-class Other than that, try to do 2 and 3 together just to test it out. There is no reason it doesn’t work. From: Todd [mailto:bit1...@163.com] Sent: Tuesday, December 22, 2015 12:01 AM To: user@hadoop.apache.org Subject: [Classpath Issue]NoClassFoundException occurs when depending on the 3rd jar Hi, I have two jars, A and B. A contains the class that has the main method, B contains the mapper and reducer. A will collect B's mapper and reducer through reflection. I am using the following commands to submit the job,but B's mapper class not found exception is thrown. 1. $HADOOP_CLASSPATH=B.jar:other jars hadoop jar A.jar 2. HADOOP_CLASSPATH=B.jar:other jars;hadoop jar A.jar 3. hadoop jar A.jar -libjars B.jar All the above three ways don't work. Can someone help me on this? Thanks!! This email and any attachments transmitted with it are intended for use by the intended recipient(s) only. If you have received this email in error, please notify the sender immediately and then delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or distribute this email without the author’s prior permission. We take precautions to minimize the risk of transmitting software viruses, but we advise you to perform your own virus checks on any attachment to this message. We cannot accept liability for any loss or damage caused by software viruses. The information contained in this communication may be confidential and may be subject to the attorney-client privilege.
Re: [Classpath Issue]NoClassFoundException occurs when depending on the 3rd jar
try: export HADOOP_CLASSPATH=B.jar hadoop jar A.jar -libjars B.jar > On Dec 22, 2015, at 1:01 AM, Todd wrote: > > Hi, > I have two jars, A and B. A contains the class that has the main method, B > contains the mapper and reducer. A will collect B's mapper and reducer > through reflection. > I am using the following commands to submit the job,but B's mapper class not > found exception is thrown. > 1. > $HADOOP_CLASSPATH=B.jar:other jars > hadoop jar A.jar > > 2. HADOOP_CLASSPATH=B.jar:other jars;hadoop jar A.jar > 3. hadoop jar A.jar -libjars B.jar > > All the above three ways don't work. Can someone help me on this? Thanks!! >
