Hi,
As a data mining developer who need to build a recommender engine POC (Proof Of
Concept) to support several future use cases, I've found Mahout framework as an
appealing place to start with. But as I'm new to Mahout and Hadoop in general
I've a couple of questions...
1. In "Mahout in action" under section 3.2.5 (Database-based data) it
says: "...Several classes in Mahout's recommender implementation will attempt
to push computations into the database for performance...". I've looked in the
documents and inside the code itself, but didn't found anywhere a reference to
what are those calculations that are pushed into the DB. Could you please
explain what could be done inside the DB?
2. My future use will include use cases with small-medium data volumes
(where I guess the non-distributed algorithms will do the job), but also use
cases that include huge amounts of data (over 500,000,000 ratings). From my
understanding this is where the distributed code should be come handy. My
question here is, because I will need to use both distributed & non-distributed
how could I build a good design here?
Should I build two different solutions on different machines? Could I do
part of the job distributed (for example similarity calculation) and the output
will be used for the non-distributed code? Is it a BKM? Also if I deploy entire
mahout code on an Hadoop environment, what does it mean for the non-distributed
code, will it all run as a different java process on the name node?
3. As for now, beside of the Hadoop cluster we are building we have some
strong SQL machines (Netezza appliance) that can handle big (structure) data
and include good integration with 3'rd party analytics providers or developing
on java platform but don't include such reach recommender framework like
Mahout. I'm trying to understand how could I utilize both solutions (Netezza &
Mahout) to handle big data recommender system use cases. Thought maybe to move
data into Netezza, do there all data manipulation and transformation, and in
the end to prepare a file that contain the classic data model structure needed
for Mahout. But could you think on better solution \ architecture? Maybe
keeping the data only inside Netezza and extracting it to Mahout using JDBC
when needed? I will be glad to hear your ideas :)
Thanks,
Oren
---------------------------------------------------------------------
Intel Electronics Ltd.
This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
