thanks for your help
please i need more explanations on these:
* it is not too far away, network-wise
what do u mean network-wise?? what are the requirements of the
connection between the client and server? because i think that my
cluster is protected with a firewall
* the client hadoop configuration is in sync with the servers
how to do this?
i have been till now only running jobs on hadoop, but i have never
configured it.
this will not mean that the client machine will be a node in the
cluster, right?
and what if my client does not have a hadoop installation and I don't
want to force him to install one just to use my tool? can't I simply
submit jobs to the cluster remotely from my java code using SSH for example?
for the file transfer, I have managed to copy files from the local file
system of the cluster to the HDFS. but I am not sure if this will work
for the local file system of the client if I configured the client
hadoop to access the cluster.
best regards ...
Steve Loughran wrote:
Deyaa Adranale wrote:
hello,
i am developing a tool that will do some analysis tasks using hadoop
map/reduce on a cluster
the tool user interfaces will be run on the client windows system and
should run the analysis tasks as map/reduce jobs on a hadoop cluster
(configured by the user).
my question is how to run hadoop jobs on a cluster from a client
machine (other than the master) from inside java code.
I know that I should have a hadoop installation on the client that
should be configured to point to the cluster's master, but I am not
sure how to do it.
you need the hadoop JARs; your client can then talk directly to a
cluster provided
* it is not too far away, network-wise
* the client hadoop configuration is in sync with the servers
You just create a JobClient instance and submit a job through it
another necessity for my tool would be to copy files from the local
client file system to the HDFS on the cluster. I am also not sure if
I can access the HDFS of the cluster from a client machine using java
code.
Yes, look in the FsShell and FileUtils classes
* None of this stuff is documented outside the source+javadocs, so you
will need to rummage around the source to work out what to do.
* Pull log4J.properties and commons-logging.properties from the hadoop
JARs if you want to route the hadoop classes logging through your own
chosen logger
