n, please return the e-mail to the sender and permanently delete
> this message and any attachments.
>
> Messages and attachments are scanned for all known viruses. Always scan
> attachments before opening them.
>
--
Regards,
*Stanley Shi,*
t; stable. So we have conclusion that the CPU utilization is related to HDFS. We
> want to know whether this issue is really related to HDFS and is there any
> solution to fix it?
>
>
>
>
>
> Thanks a lot!
>
>
>
> BR/Shiyuan
>
--
Regards,
*Stanley Shi,*
>>>>> This will display all the information you need about the blocks of
>>>>> your file.
>>>>>
>>>>> Hope it helps.
>>>>> Yehia
>>>>>
>>>>>
>>>>> On 27 August 2014 20:18, Demai Ni
on
> port 50070, you can't view a file in the browser. Clicking a file gives a
> little popup with metadata and a download link. Can HDFS be configured to
> show plaintext file contents in the browser?
>
> Thanks,
> Brian
>
>
--
Regards,
*Stanley Shi,*
h pt = new Path("${hdfsName}/${dir}/${outFile}")
>
> def fs = org.apache.hadoop.fs.FileSystem.get(configuration);
>
> FSDataOutputStream fp = fs.create(pt, true)
>
> fp << "${key} ${value}\n"
> On 27 Aug 2014 09:46, "Stanley Shi" wrote:
>
>> woul
ubmit a map reduce job, would it only work on
> the files present at that point ??
>
>
>
> Regards,
> Nikhil Kandoi
>
>
>
--
Regards,
*Stanley Shi,*
t; On 27 August 2014 20:18, Demai Ni wrote:
>>>
>>>> Hi, Stanley,
>>>>
>>>> Many thanks. Your method works. For now, I can have two steps approach:
>>>> 1) getFileBlockLocations to grab hdfs BlockLocation[]
>>>> 2) use l
is not done in single attempt. The file needs to be
>>> appended with streams of information generated.
>>>
>>> In the code, inside a loop, I open a file in hdfs, appends some
>>> information. This is not working and I see only the last write.
>>>
>>> How do I accomplish append operation in hadoop? Can anyone share a
>>> pointer to me?
>>>
>>>
>>>
>>>
>>> regards
>>> Bala
>>>
>>
>>
--
Regards,
*Stanley Shi,*
h as: repl =3[
> /rack/hdfs01, /rack/hdfs02...]
>
> With such info, is there a way to
> 1) login to hfds01, and read the block directly at local file system level?
>
>
> Thanks
>
> Demai on the run
--
Regards,
*Stanley Shi,*
l impossible to get out of the safe mode and as log as we are in safe
> mode we can't delete anything in the HDFS.
>
>
> Is anyone having a tip about this issue?
>
>
> Thankfully,
>
> Vincent.
>
>
>
--
Regards,
*Stanley Shi,*
Your client side was running at "14/07/24 18:35:58 INFO mapreduce.Job:
T***", But you are pasting NN log at "2014-07-24 17:39:34,255";
By the way, which version of HDFS are you using?
Regards,
*Stanley Shi,*
On Fri, Jul 25, 2014 at 10:36 AM, ch huang wrote:
> 2014-
The "alex" should belongs to "hadoop" group on namenode;
Regards,
*Stanley Shi,*
On Thu, Jul 24, 2014 at 10:11 PM, java8964 wrote:
> Are you sure user 'Alex' belongs to 'hadoop' group? Why not your run
> command 'id alex' to prove it?
Would you please also past the corresponding namenode log?
Regards,
*Stanley Shi,*
On Fri, Jul 25, 2014 at 9:15 AM, ch huang wrote:
> hi,maillist:
>i try to copy data from my old cluster to new cluster,i get
> error ,how to handle this?
>
> 14/07/24 18:35:58 INF
Do you have a secondary namenode running? Secondary NN is used for this
purpose;
Also, if you have HDFS HA enabled, this problem will also not occur.
Regards,
*Stanley Shi,*
On Tue, Jul 22, 2014 at 7:24 AM, Anfernee Xu wrote:
> Hi,
>
> For some reason, all PIDs file are missing in m
which distribution are you using?
Regards,
*Stanley Shi,*
On Thu, Jul 24, 2014 at 4:38 AM, andrew touchet wrote:
> I should have added this in my first email but I do get an error in the
> data node's log file
>
> '2014-0
There's a DistCP utility for this kind of purpose;
Also there's "Spring XD" there, but I am not sure if you want to use it.
Regards,
*Stanley Shi,*
On Mon, Jul 7, 2014 at 10:02 PM, Mohan Radhakrishnan <
radhakrishnan.mo...@gmail.com> wrote:
> Hi,
>
will report that blocks are missing;
1. Since you're using HA, lots of the editlogs are stored in the journal
node; the fsimage you have may not be exactly the one you want;
Regards,
*Stanley Shi,*
On Tue, Jul 8, 2014 at 8:12 AM, cho ju il wrote:
>
>
> Thank
You can get the wikipedia data from it's website, it's pretty big;
Regards,
*Stanley Shi,*
On Tue, Jul 8, 2014 at 1:35 PM, Du Lam wrote:
> Configuration conf = getConf();
> conf.setLong("mapreduce.input.fileinputformat.split.maxsize",1000);
>
> // u can
It will start from scratch to copy all map outputs from all mapper nodes;
Regards,
*Stanley Shi,*
On Thu, Jul 3, 2014 at 2:28 PM, James Teng wrote:
> First i would like to declare that although i am not new to hadoop, but
> not expert on it as well.
> i would like to consult one
The "similar" logic is not transitive, that means, if a is similar to b, b
is similar to c, but a may be not similar to c;
then how do you do the group?
Regards,
*Stanley Shi,*
On Sat, Jun 21, 2014 at 2:51 AM, parnab kumar wrote:
> Hi,
>
> I have a set of hashes. Ea
Cluster-running upgrade is only supported after 2.4, that is, upgrade from
2.4 to 2.4+ is supported; upgrading from 2.2 to 2.4 is not supported;
Regards,
*Stanley Shi,*
On Fri, Jun 20, 2014 at 5:50 PM, Jason Meng wrote:
> Hi,
>
> I setup Hadoop 2.2 cluster with NameNode HA. How to u
should set it on RM node;
Regards,
*Stanley Shi,*
On Wed, Jun 4, 2014 at 9:24 AM, ch huang wrote:
> hi,maillist:
> i installed my job history server on my one of NN(i use NN HA)
> ,i want to ask if i need set history server address on each node?
>
Another possible reason is that you are not using the correct conf file;
Regards,
*Stanley Shi,*
On Tue, Jun 3, 2014 at 6:53 AM, Rajat Jain wrote:
> Have you tried setting fs.defaultFS with the same value?
>
>
> On Sat, May 31, 2014 at 11:22 AM, ishan patwa
> wrote:
>
>
Regards,
*Stanley Shi,*
On Thu, May 22, 2014 at 10:46 AM, Stanley Shi wrote:
> seems my guess is correct; I mean in your program, you can call:
>
>
> *hadoop jar myjar.jar input output " " *
>
> *instead you use:*
>
> *hadoop jar myjar.jar input output
Are you trying to pass arguments from user input? reading input from stdin?
I suggest you use some special characters to express; for example, let
args[2] == "TAB" and in your program, you can convert this "TAB" to the
real delimeter you want to use ("\t")
Regards,
instead it will wait
for some other hosts to register as the short name (which will not happen);
Regards,
*Stanley Shi,*
On Mon, May 19, 2014 at 10:24 PM, REYANE OUKPEDJO wrote:
> There seems to be an issue with the container request when specifying the
> host name for the container. W
the port 8088 is the new web ui for yarn; it is supported by the resource
manager process;
Regards,
*Stanley Shi,*
On Fri, May 16, 2014 at 8:14 AM, Software Dev wrote:
> It's been awhile since playing with Hadoop. Recently installed the
> latest version on my mac via homebrew and
Would you please explain more on what you are trying to do?
Perhaps Distcp is a good example for you.
Regards,
*Stanley Shi,*
On Wed, May 14, 2014 at 1:09 PM, Dhrumil Shah wrote:
> Hi
>
>
> Can you please let me know how can i read multiple file from mulitple
> location in m
Put the configuration files in classpath before the default ones;
Regards,
*Stanley Shi,*
On Thu, May 15, 2014 at 6:30 AM, Software Dev wrote:
> I am trying to run a jar from my command line and it keeps picking up
> the default hadoop configuration values for core,hdfs,mapred, etc.
>
hadoop-daemon stop nodemanager??
Regards,
*Stanley Shi,*
On Wed, May 7, 2014 at 11:48 AM, Henry Hung wrote:
> Is there a way to gracefully stop a nodemanager?
>
>
>
> I want to make some yarn-site.xml and mapred-site.xml configuration
> changes, and need to resta
please check you configuration files, are there anywhere mentioning
"localhost"? "localhost" should not be used if you are deploying an
distributed cluster.
Regards,
*Stanley Shi,*
On Tue, May 13, 2014 at 6:52 PM, Gäde, Sebastian wrote:
> Hi,
>
> I've set
The "FileNotFoundException" doesn't mean anything in the pi program. If you
have some error and the program didn't run successfully, it will always
throw this exception.
What do you have in the opts?
Regards,
*Stanley Shi,*
On Mon, May 12, 2014 at 2:09 PM, Tao
Yes, why not?
Regards,
*Stanley Shi,*
On Sun, May 11, 2014 at 9:57 PM, Karim Awara wrote:
> Hi,
>
> Can I open multiple files on hdfs and write data to them in parallel and
> then close them at the end?
>
> --
> Best Regards,
> Karim Ahmed Awara
>
> ---
in some case you may not find the third node to place replica.
Regards,
*Stanley Shi,*
On Sun, May 11, 2014 at 10:55 AM, jianan hu wrote:
> Hi everyone,
>
> See HDFS documents, It says "For the common case, when the replication
> factor is three, HDFS’s placement policy is t
Please be sure to use different HADOOP_CONF_DIR for the two version; and
also in the configuration, be sure to use different folder to store the
HDFS related files;
Regards,
*Stanley Shi,*
On Tue, May 6, 2014 at 8:41 AM, Shengjun Xin wrote:
> According to your description, I think it
NoRouteToHost, please check your network setting
Regards,
*Stanley Shi,*
On Fri, Apr 18, 2014 at 3:42 PM, wrote:
> Hi,
>
>
>
> No errors in hdfsConnect().
>
> But if I call hdfsCreateDirectory() after hdfsConnect() , got errors as
> followed:
>
>
>
> hd
...
Regards,
*Stanley Shi,*
On Fri, Apr 18, 2014 at 7:42 AM, david marion wrote:
> I'm having an issue in client code where there are multiple clusters with
> HA namenodes involved. Example setup using Hadoop 2.3.0:
>
> Cluster A with the following properties defined i
Is this what you are looking for?
http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/CommandsManual.html#daemonlog
Regards,
*Stanley Shi,*
On Wed, Apr 16, 2014 at 2:06 AM, Ashwin Shankar
wrote:
> Thanks Gordon and Stanley, but this would require us to bounce the proc
can do you an "unzip -l myjob.jar" to see if your jar file has the correct
hierarchy?
Regards,
*Stanley Shi,*
On Tue, Apr 15, 2014 at 6:53 PM, laozh...@sina.cn wrote:
> Thank you for your advice . When i user your command , i get the below
> error info .
> $ hadoop jar my
You can just follow any instruction on deploying distributed cluster, just
put several different services on the same host;
Regards,
*Stanley Shi,*
On Tue, Apr 15, 2014 at 12:02 PM, Mohan Radhakrishnan <
radhakrishnan.mo...@gmail.com> wrote:
> Hi,
> I have 2 nodes, one is
it could be just to analyze the above size of data.
>
> Regards
> Shashidhar
>
--
Regards,
*Stanley Shi,*
; Thanks,
> Ashwin
>
>
>
--
Regards,
*Stanley Shi,*
gt;
>
> 2014-04-14 17:04 GMT+08:00 Stanley Shi
>
> >:
>
>> Please find it in this page: https://wiki.apache.org/hadoop/Roadmap
>>
>> hadoop 2.3.0 only include "phase 1" of the heterogeneous storage; "phase
>> 2" will be included in 2.5.0
Please find it in this page: https://wiki.apache.org/hadoop/Roadmap
hadoop 2.3.0 only include "phase 1" of the heterogeneous storage; "phase 2"
will be included in 2.5.0;
Regards,
*Stanley Shi,*
On Mon, Apr 14, 2014 at 4:38 PM, ascot.m...@gmail.com
wrote:
> hi,
>
I believe there's some command to show list of datanodes from CLI, using
parsing HTML is not a good idea. HTML page is intended to be read by human.
I also don't know how to refresh node list;
Regards,
*Stanley Shi,*
On Sat, Apr 12, 2014 at 11:31 AM, Ashwin Shankar
wrote:
> Hi,
do you any node manager running?
Regards,
*Stanley Shi,*
On Mon, Apr 14, 2014 at 11:37 AM, Rahul Singh wrote:
> Hi,
> I am trying to run wordcount example(input file contains just few words)
> but the job seems to be stuck. How do i debug what went wrong?
>
> [hduser@po
Not sure if this helps but copyFromLocal is just to write data from the
current client machine to hdfs; but distcp will start a mapreduce job to do
the copy, that means the NodeManager/taskTracker machine need to write data
to the remote hdfs cluster;
Regards,
*Stanley Shi,*
On Sun, Apr 13
AFAIK, no tools now.
Regards,
*Stanley Shi,*
On Fri, Apr 11, 2014 at 9:09 AM, ch huang wrote:
> hi,maillist:
> how can i archive old data in HDFS ,i have lot of old data ,the
> data will not be use ,but it take lot of space to store it ,i want to
> archive and zip the ol
AFAIK, data node will be dead since it cannot handle one disk failure ( if
it cannot write data to a disk, it will fail).
But not sure if this situation has changed.
Regards,
*Stanley Shi,*
On Thu, Apr 3, 2014 at 7:29 PM, Bharath Kumar wrote:
>
> Hi All,
> I have a q
This doesn't seem like related with the data size.
How much memory do you use for the reducer?
Regards,
*Stanley Shi,*
On Thu, Apr 3, 2014 at 8:04 AM, Li Li wrote:
> I have a map reduce program that do some matrix operations. in the
> reducer, it will average many large matrix(
what do you mean by "document"? are you looking for this?
http://hadoop.apache.org/docs/r2.2.0/api/index.html
Regards,
*Stanley Shi,*
On Thu, Apr 3, 2014 at 10:53 AM, EdwardKing wrote:
> I look through http://hadoop.apache.org/docs/r2.2.0/ ,but I don't find
> any url
map task number is not decided by the resources you need.
It's decided by something else.
Regards,
*Stanley Shi,*
On Wed, Apr 2, 2014 at 9:08 AM, Libo Yu wrote:
> Hi all,
>
> I pretty much use the default yarn setting to run a word count example on
> a 3 node cluster. Her
FileSystem.getFileBlockLocations(...)
Regards,
*Stanley Shi,*
On Fri, Mar 28, 2014 at 10:03 AM, Libo Yu wrote:
> Hi all,
>
> "hadoop path fsck -files -block -locations" can list locations for all
> blocks in the path.
> Is it possible to list all blocks and the bl
where did you get the error? from the compiler or the runtime?
Regards,
*Stanley Shi,*
On Thu, Mar 27, 2014 at 7:34 AM, Jonathan Poon wrote:
> Hi Everyone,
>
> I'm submitting a MapReduce job using the -files option to copy a text file
> that contains properties I use for t
ou can use the class
"org.apache.hadoop.mapreduce.lib.input.TextInputFormat" to read each record
from you output file.
Regards,
*Stanley Shi,*
On Tue, Mar 25, 2014 at 9:07 PM, Li Li wrote:
> I have map-reduce job to output my custom writable objects, how can I
> read it using pure java api?
> I don't want to serial
seems like a vendor specific problem. Ask this in HDP forum.
Regards,
*Stanley Shi,*
On Mon, Mar 24, 2014 at 9:50 AM, Anfernee Xu wrote:
> Hi,
>
> All dataNode in my cluster failed to start due to below error
>
> notice:
> /Stage[2]/Hdp-hadoop::Datanode/Hdp-hadoop::Serv
Can you confirm that you namenode image and fseditlog are still there? if
not, then your data IS lost.
Regards,
*Stanley Shi,*
On Sun, Mar 23, 2014 at 6:24 PM, Fatih Haltas wrote:
> No, not ofcourse I blinded it.
>
>
> On Wed, Mar 19, 2014 at 5:09 PM, praveenesh kumar wrote:
pp.getName() + " " + itr.nextToken());
context.write(word, one);
}
}
}
Note: add your filtering code here;
and then when running the command, use you input path as param;
Regards,
*Stanley Shi,*
On Fri, Mar 21, 2014 at 9:32 AM, Stanley Shi wrote:
> Just reviewed
Just reviewed the code again, you are not really using map-reduce. you are
reading all files in one map process, this is not a normal map-reduce job
works.
Regards,
*Stanley Shi,*
On Thu, Mar 20, 2014 at 1:50 PM, Ranjini Rathinam wrote:
> Hi,
>
> If we give the b
rd,one);
Regards,
*Stanley Shi,*
On Wed, Mar 19, 2014 at 8:50 PM, Ranjini Rathinam wrote:
> Hi,
>
> I have folder named INPUT.
>
> Inside INPUT i have 5 resume are there.
>
> hduser@localhost:~/Ranjini$ hadoop fs -ls /user/hduser/INPUT
> Found 5 items
> -rw-r--r--
server/namenode/BlockPlacementPolicyWithNodeGroup.java
--If I want to control the block placement then I have to write codes
rather than type shell commands?
if you want to implement your own logic on block placement, you have to
write code.
Regards,
*Stanley Shi,*
On Wed, Mar 19, 2014 at 3:07 AM, E
Ah yes, I overlooked this. Then please check the file are there or not: "ls
/home/hadoop/project/hadoop-data/dfs/name"?
Regards,
*Stanley Shi,*
On Tue, Mar 18, 2014 at 2:06 PM, Azuryy Yu wrote:
> I don't think this is the case, because there is;
>
> hadoop.tm
me.dir
${hadoop.tmp.dir}/dfs/name
Determines where on the local filesystem the DFS name node
should store the name table(fsimage). If this is a comma-delimited list
of directories then the name table is replicated in all of the
directories, for redundancy.
*
Regards,
*S
2>
";
If you want to use it, you need to write your own policy, please see this
JIRA for example: https://issues.apache.org/jira/browse/HDFS-3601
Regards,
*Stanley Shi,*
On Mon, Mar 17, 2014 at 11:31 AM, Eric Chiu wrote:
> HI all,
>
> Could anyone tell me How to install a
download it, unzip and put it back?
Regards,
*Stanley Shi,*
On Fri, Mar 14, 2014 at 5:44 PM, Sai Sai wrote:
> Can some one please help:
> How to unzip a .tar.bz2 file which is in hadoop/hdfs
> Thanks
> Sai
>
>
For reading PDF in java, you may refer to this link:
http://stackoverflow.com/questions/4784825/how-to-read-pdf-files-using-java
in mapreduce, you can use the same code; except that each map() function
processes one file;
Regards,
*Stanley Shi,*
On Wed, Mar 12, 2014 at 4:53 PM, Ranjini
There's no limitation on same-subnet.
Regards,
*Stanley Shi,*
On Wed, Mar 12, 2014 at 1:31 PM, navaz wrote:
> Hi all
>
> Question regarding hadoop architecture . Generally in hadoop cluster nodes
> are placed in racks and all the nodes connected to top of the rack switch
just start the new node with the same configuration as in namenode, after
sometime, you will see the new node list.
Regards,
*Stanley Shi,*
On Tue, Mar 11, 2014 at 9:07 AM, Parmeet wrote:
> Hello,
>
> I am trying to add a new dataNode to existing hadoop cluster & you would
&
Waiting for others to give best practice.
I think you can use eclipse to manage the maven; see the full dependency
hierarchy, if some jar(for example, guava) exists in both hadoop dependency
chain and your own requirements, put your requirements' scope as "provided"
.
Regard
You can read from http://resource-manager.host.ip:8088/conf
This is an xml format file you can use directly.
Regards,
*Stanley Shi,*
On Fri, Mar 7, 2014 at 1:46 AM, John Lilley wrote:
> How would I go about fetching configuration values (e.g. yarn-site.xml)
> from the cluster via t
May be you can try download the LZO class and rebuild it against Hadoop
2.2.0;
If build success, you should be good to go;
if failed, then maybe you need to wait for the LZO guys to update their
code.
Regards,
*Stanley Shi,*
On Thu, Mar 6, 2014 at 6:29 PM, KingDavies wrote:
> Running
Maybe your console and browser are using different settings, would you
please try "wget
http://repo.maven.apache.org/maven2/org/apache/felix/maven-bundle-plugin/2.4.0/maven-bundle-plugin-2.4.0.pom
"?
Regards,
*Stanley Shi,*
On Wed, Mar 5, 2014 at 6:59 PM, Avinash Kujur wrote:
which version of hadoop you are using?
This is something similar with your error log:
http://stackoverflow.com/questions/19895969/can-access-hadoop-fs-through-shell-but-not-through-java-main
Regards,
*Stanley Shi,*
On Wed, Mar 5, 2014 at 4:29 PM, 张超 wrote:
> Hi all，
> Here is a proble
Which version of hadoop are you using?
There's a possibility that the hadoop environment already have a avro**.jar
in place, thus caused the jar conflict.
Regards,
*Stanley Shi,*
On Tue, Mar 4, 2014 at 11:25 PM, John Pauley wrote:
> Outside hadoop: avro-1.7.6
> Inside hadoop:
Why you have 2 hadoop version in the same pom file? In this case, you are
not going to know which hadoop class you are actually using.
org.apache.hadoop
hadoop-client
2.3.0
org.apache.hadoop
hadoop-core
1.2.1
Regards,
*Stanley Shi
which avro version are you using when running outside of hadoop?
Regards,
*Stanley Shi,*
On Mon, Mar 3, 2014 at 11:49 PM, John Pauley wrote:
> This is cross posted to avro-user list (
> http://mail-archives.apache.org/mod_mbox/avro-user/201402.mbox/%3ccf3612f6.94d2%25jo
In Hadoop 2.2, there's no actual jobtracker running, you may want to access
the Resource Manager Web UI: http://172.11.12.6:8088/
Regards,
*Stanley Shi,*
On Mon, Mar 3, 2014 at 2:07 PM, EdwardKing wrote:
> I use Hadoop 2.2 and I want to run MapReduce web UI,so I visit followi
77 matches
Mail list logo
