[
https://issues.apache.org/jira/browse/BIGTOP-1272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14099380#comment-14099380
]
jay vyas edited comment on BIGTOP-1272 at 8/16/14 5:01 PM:
-----------------------------------------------------------
Okay bhashit. I built it succesfull. w/ jdk 7 (not 6)... Lets make sure we
add this stuff to readme:
- Java 1.7 is required.
- gradle 2.0 is (i assume) required
... Now ill test it on a cluster and let you know......
- First we need to export HADOOP_CLASSPATH=pig.jar:mahout.jar (so local client
has access to libs)
- THEN you also need to append -libjars ${JARS} (so mappers have access to the
stuff)
- THEN ...... *IT WORKS* :)
*Now please do the following so we can commit !*
- Remove the added trailing whitespace in the files under bigtop-bigpetstore/
- update the README with the following directions for running:
{noformat}
### Note that both pig and mahout can be yum installed
### via bigtop. Mahout 2.0 can also be yum installed from
### any vendor distro. you dont need to build those jars.
### As usual, generate the data.
hadoop jar bigpetstore.jar org.apache.bigtop.bigpetstore.generator.BPSGenerator
100 bigpetstore/gen
# For yarn node managers that run the actual tasks, we need mahout/pig on cp.
export
JARS="/usr/lib/pig/pig-0.12.0-.....1.0-withouthadoop.jar,/usr/lib/mahout/mahout-core-job.jar"
# For the client, we also need these jars on the cp to kick off the jobs.
export HADOOP_CLASSPATH=`echo $JARS | sed s/,/:/g`
### Now, clean it with pig.
hadoop jar bps.jar org.apache.bigtop.bigpetstore.etl.PigCSVCleaner -libjars
$JARS bigpetstore/gen/ bigpetstore/pig/ BPS_analytics.pig
### Finally, process with mahout.
hadoop jar bps.jar org.apache.bigtop.bigpetstore.recommend.ItemRecommender
-libjars $JARS,/usr/lib/mahout/mahout-core-job.jar bigpetstore/pig/Mahout
bigpetstore/Mahout/AlsFactorization bigpetstore/Mahout/AlsRecommendations
{noformat}
After you make those 2 very minor modifications, I can commit this. *THANKS*
for sticking with me through all this testing.
was (Author: jayunit100):
Okay bhashit. I built it succesfull. w/ jdk 7 (not 6)... Lets make sure we
add this stuff to readme:
- Java 1.7 is required.
- gradle 2.0 is (i assume) required
... Now ill test it on a cluster and let you know......
- First we need to export HADOOP_CLASSPATH=pig.jar:mahout.jar (so local client
has access to libs)
- THEN you also need to append -libjars ${JARS} (so mappers have access to the
stuff)
- THEN ...... *IT WORKS* :)
*Now please do the following so we can commit !*
- Remove the added trailing whitespace in the files under bigtop-bigpetstore/
- update the README with the following directions for running:
{noformat}
hadoop jar bigpetstore.jar org.apache.bigtop.bigpetstore.generator.BPSGenerator
100 bigpetstore/gen
export
JARS="/usr/lib/pig/pig-0.12.0-.....1.0-withouthadoop.jar,/usr/lib/mahout/mahout-core-0.9-...0-job.jar"
export HADOOP_CLASSPATH=`echo $JARS | sed s/,/:/g`
hadoop jar bps.jar org.apache.bigtop.bigpetstore.etl.PigCSVCleaner -libjars
$JARS bigpetstore/gen/ bigpetstore/pig/ BPS_analytics.pig
hadoop jar bps.jar org.apache.bigtop.bigpetstore.recommend.ItemRecommender
-libjars $JARS,/usr/lib/mahout/mahout-core-0.9-cdh5.1.0-job.jar
bigpetstore/pig/Mahout bigpetstore/Mahout/AlsFactorization
bigpetstore/Mahout/AlsRecommendations
{noformat}
After you make those 2 very minor modifications, I can commit this. *THANKS*
for sticking with me through all this testing.
> BigPetStore: Productionize the Mahout recommender
> -------------------------------------------------
>
> Key: BIGTOP-1272
> URL: https://issues.apache.org/jira/browse/BIGTOP-1272
> Project: Bigtop
> Issue Type: New Feature
> Components: blueprints
> Affects Versions: backlog
> Reporter: jay vyas
> Attachments: BIGTOP-1272.patch, BIGTOP-1272.patch, BIGTOP-1272.patch,
> BIGTOP-1272.patch, BIGTOP-1272.patch, arch.jpeg, build.gradle
>
>
> BIGTOP-1271 adds patterns into the data that gaurantee that a meaningfull
> type of product recommendation can be given for at least *some* customers,
> since we know that there are going to be many customers who only bought 1
> product, and also customers that bought 2 or more products -- even in a
> dataset size of 10. due to the gaussian distribution of purchases that is
> also in the dataset generator.
> The current mahout recommender code is statically valid: It runs to
> completion in local unit tests if a hadoop 1x tarball is present but
> otherwise it hasn't been tested at scale. So, lets get it working. this
> JIRA also will comprise:
> - deciding wether to use mahout 2x for unit tests (default on mahout maven
> repo is the 1x impl) and wether or not bigtop should host a mahout 2x jar?
> After all, bigtop builds a mahout 2x jar as part of its packaging process,
> and BigPetStore might thus need a mahout 2x jar in order to test against the
> right same of bigtop releases.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
