On 10/9/2011 7:56 PM, Amindri Udugala wrote:
Also I'm interested on the topic "Ability to dynamically increase replicas
of data in HDFS based on access patterns". I saw that it is reported as an
issue at jira https://issues.apache.org/jira/browse/HDFS-782 . The
resolution is still marked as unresolved. I would like to know if any work
is currently carried out about this.
Regarding this, we are currently investigating some models (based on a
Matlab prototype that utilizes geneticaly optimized artificial neural
networks) that have the goal of performing time series analysis of the
access patterns. What we do is that we take the access logs, extract the
average bytes/sec reads and writes in x-minute intervals (may be
whatever one desires, we used 15-minute intervals) thus creating two
time series, one for read operations and one for write operations with
these average values. Then by utilizing the time series prediction model
we can look into the future and decide proactively which replication
factors to change.
You can find the paper referring to this technique here:
http://users.ntua.gr/gkousiou/publications/PID2095917.pdf
We began this approach for a slightly different reason (choosing the
fittest service for federation in a federated cloud scenario), but also
for adjusting the replication factor (which is a simple HDFS command).
At this stage it is just a matlab prototype (however it can be used
online for example by using the following approach:
http://www.computer.org/portal/web/csdl/doi/10.1109/SCC.2010.37
accessible also here
http://users.ntua.gr/gkousiou/publications/SCC2010.pdf )
However we are planning in the future to create the models through
Apache Mahout project, in order to fully exploit the capabilities of
Hadoop and MapReduce.
BR,
George
--
---------------------------
George Kousiouris
Electrical and Computer Engineer
Division of Communications,
Electronics and Information Engineering
School of Electrical and Computer Engineering
Tel: +30 210 772 2546
Mobile: +30 6939354121
Fax: +30 210 772 2569
Email: gkous...@mail.ntua.gr
Site: http://users.ntua.gr/gkousiou/
National Technical University of Athens
9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
