[
https://issues.apache.org/jira/browse/BIGTOP-1272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14099380#comment-14099380
]
jay vyas edited comment on BIGTOP-1272 at 8/16/14 2:01 AM:
-----------------------------------------------------------
Okay bhashit. I built it succesfull. w/ jdk 7 (not 6)... Lets make sure we
add this stuff to readme:
- Java 1.7 is required.
- gradle 2.0 is (i assume) required
... Now ill test it on a cluster and let you know......
- First we need to export HADOOP_CLASSPATH=pig.jar:mahout.jar (so local client
has access to libs)
- THEN you also need to append -libjars ${JARS} (so mappers have access to the
stuff)
- THEN ...... *IT WORKS* :)
Here is how to run it. Please update the readme accordingly, and also cleanup
the whitespace (there are alot of whitespace errors, you can test by applying
the patch cleanly).
Thanks bhashit. this is a huge patch and an uber-modern hadoop application
template.
Now please do the following so we can commit !
- Remove the added trailing whitespace in the files under bigtop-bigpetstore/
- update the README with the following directions
{noformat}
hadoop jar bigpetstore.jar org.apache.bigtop.bigpetstore.generator.BPSGenerator
100 bigpetstore/gen
export
JARS="/usr/lib/pig/pig-0.12.0-.....1.0-withouthadoop.jar,/usr/lib/mahout/mahout-core-0.9-...0-job.jar"
export HADOOP_CLASSPATH=`echo ${JARS} | sed s/,/:/g`
hadoop jar bps.jar org.apache.bigtop.bigpetstore.etl.PigCSVCleaner -libjars
${JARS} bigpetstore/gen/ bigpetstore/pig/ BPS_analytics.pig
hadoop jar bps.jar org.apache.bigtop.bigpetstore.recommend.ItemRecommender
-libjars ${JARS},/usr/lib/mahout/mahout-core-0.9-cdh5.1.0-job.jar
bigpetstore/pig/Mahout bigpetstore/Mahout/AlsFactorization
bigpetstore/Mahout/AlsRecommendations
{noformat}
After you make those 2 very minor modifications, I can commit this.
*THANKS* for sticking with me through all this testing.
was (Author: jayunit100):
Okay bhashit. I built it succesfull. w/ jdk 7 (not 6)... Lets make sure we
add this stuff to readme:
- Java 1.7 is required.
- gradle 2.0 is (i assume) required
... Now ill test it on a cluster and let you know......
- First we need to export HADOOP_CLASSPATH=pig.jar:mahout.jar (so local client
has access to libs)
- THEN you also need to append -libjars ${JARS} (so mappers have access to the
stuff)
- THEN ...... *IT WORKS* :)
Here is how to run it. Please update the readme accordingly, and also cleanup
the whitespace (there are alot of whitespace errors, you can test by applying
the patch cleanly).
Thanks bhashit. this is a huge patch and an uber-modern hadoop application
template.
Here are the results.
{noformat}
CT litter_box 2
...
CT salmon_bait 2
NY fish_bowl 1
...
OK dog_house 1
..
OK leather_collar 1
[root@localhost vagrant]# hadoop fs -ls /user/hdfs/bigpetstore
Found 4 items
drwxr-xr-x - hdfs supergroup 0 2014-08-16 01:49
/user/hdfs/bigpetstore/Mahout
drwxr-xr-x - hdfs supergroup 0 2014-08-16 01:22
/user/hdfs/bigpetstore/gen
drwxr-xr-x - hdfs supergroup 0 2014-08-16 01:40
/user/hdfs/bigpetstore/pig
drwxr-xr-x - hdfs supergroup 0 2014-08-16 01:41
/user/hdfs/bigpetstore/pig_ad_hoc_script0
[root@localhost vagrant]# hadoop fs -cat
/user/hdfs/bigpetstore/Mahout/AlsRecommendations/*
2 [19:1.0]
...
92 [19:1.0]
{noformat}
Now please do the following so we can commit !
- Remove the added trailing whitespace in the files under bigtop-bigpetstore/
- update the README with the following directions:
{noformat}
hadoop jar bigpetstore.jar org.apache.bigtop.bigpetstore.generator.BPSGenerator
100 bigpetstore/gen
export
JARS="/usr/lib/pig/pig-0.12.0-.....1.0-withouthadoop.jar,/usr/lib/mahout/mahout-core-0.9-...0-job.jar"
export HADOOP_CLASSPATH=`echo ${JARS} | sed s/,/:/g`
hadoop jar bps.jar org.apache.bigtop.bigpetstore.etl.PigCSVCleaner -libjars
${JARS} bigpetstore/gen/ bigpetstore/pig/ BPS_analytics.pig
hadoop jar bps.jar org.apache.bigtop.bigpetstore.recommend.ItemRecommender
-libjars ${JARS},/usr/lib/mahout/mahout-core-0.9-cdh5.1.0-job.jar
bigpetstore/pig/Mahout bigpetstore/Mahout/AlsFactorization
bigpetstore/Mahout/AlsRecommendations
{noformat}
After you make those 2 very minor modifications, I can commit this.
*THANKS* for sticking with me through all this testing.
> BigPetStore: Productionize the Mahout recommender
> -------------------------------------------------
>
> Key: BIGTOP-1272
> URL: https://issues.apache.org/jira/browse/BIGTOP-1272
> Project: Bigtop
> Issue Type: New Feature
> Components: blueprints
> Affects Versions: backlog
> Reporter: jay vyas
> Attachments: BIGTOP-1272.patch, BIGTOP-1272.patch, BIGTOP-1272.patch,
> BIGTOP-1272.patch, BIGTOP-1272.patch, arch.jpeg, build.gradle
>
>
> BIGTOP-1271 adds patterns into the data that gaurantee that a meaningfull
> type of product recommendation can be given for at least *some* customers,
> since we know that there are going to be many customers who only bought 1
> product, and also customers that bought 2 or more products -- even in a
> dataset size of 10. due to the gaussian distribution of purchases that is
> also in the dataset generator.
> The current mahout recommender code is statically valid: It runs to
> completion in local unit tests if a hadoop 1x tarball is present but
> otherwise it hasn't been tested at scale. So, lets get it working. this
> JIRA also will comprise:
> - deciding wether to use mahout 2x for unit tests (default on mahout maven
> repo is the 1x impl) and wether or not bigtop should host a mahout 2x jar?
> After all, bigtop builds a mahout 2x jar as part of its packaging process,
> and BigPetStore might thus need a mahout 2x jar in order to test against the
> right same of bigtop releases.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
