Hi, i am a beginner too, but as i have learned, hadoop works better with
big files, at least with 64MB, 128MB or even more. I think you need to
aggregate all the files into a new big one. Then you must copy to HDFS
using this command:
hadoop fs -put MYFILE /YOUR_ROUTE_ON_HDFS/MYFILE
hadoop just copy MYFILE into hadoop distributed file system.
Can i recommend you what i have done? go to BigDataUniversity.com and take
the Hadoop Fundamentals I course. It is free and very well documented.
Regards
Alonso Isidoro Roman.
Mis citas preferidas (de hoy) :
Si depurar es el proceso de quitar los errores de software, entonces
programar debe ser el proceso de introducirlos...
- Edsger Dijkstra
My favorite quotes (today):
If debugging is the process of removing software bugs, then programming
must be the process of putting ...
- Edsger Dijkstra
If you pay peanuts you get monkeys
2014-03-03 12:10 GMT+01:00 goi cto goi@gmail.com:
Hi,
I am sorry for the beginners question but...
I have a spark java code which reads a file (c:\my-input.csv) process it
and writes an output file (my-output.csv)
Now I want to run it on Hadoop in a distributed environment
1) My inlut file should be one big file or separate smaller files?
2) if we are using smaller files, how does my code needs to change to
process all of the input files?
Will Hadoop just copy the files to different servers or will it also split
their content among servers?
Any example will be great!
--
Eran | CTO
