[
https://issues.apache.org/jira/browse/BIGTOP-1366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14214131#comment-14214131
]
jay vyas edited comment on BIGTOP-1366 at 11/16/14 11:33 PM:
-------------------------------------------------------------
[~rnowling] okay thanks for this.
- It looks like you need to update {{build.gradle}} in the restructuring, to
reference "../../pom.xml" instead of "../pom.xml". BigPetStore builds pull
some data in from bigtop pom , by default, in from top level bigtop. Easy fix
.
- Also , you have a lot of trailing whitespaces. I can fix this on commit via
{{--fix-whitespace}} so its not a huge problem.
- Can you add a gradle test that launches a local spark job. Right now there
are none. ill try to paste a snippet of how to do this tonite if i can
- otherwise, looks like the existing map reduce code still works, and the spark
code looks good as well !
FYI
{noformat}
org.apache.bigtop.bigpetstore.docs.TestDocs > testGraphViz PASSED
org.apache.bigtop.bigpetstore.generator.TestNumericalIdUtils > testName PASSED
org.apache.bigtop.bigpetstore.generator.TestPetStoreTransactionGeneratorJob >
test PASSED
BUILD SUCCESSFUL
Total time: 2 mins 3.84 secs
{noformat}
was (Author: jayunit100):
[~rnowling] okay thanks for this.
- It looks like you need to update {{build.gradle}} in the restructuring, to
reference "../../pom.xml" instead of "../pom.xml". BigPetStore builds pull
some data in from bigtop pom , by default, in from top level bigtop. Easy fix
.
- Also , you have a lot of trailing whitespaces. I can fix this on commit via
{{--fix-whitespace}} so its not a huge problem.
- otherwise, looks like the existing map reduce code still works, and the spark
code looks good as well !
> Updated, Richer Model for Generating Data for BigPetStore
> ----------------------------------------------------------
>
> Key: BIGTOP-1366
> URL: https://issues.apache.org/jira/browse/BIGTOP-1366
> Project: Bigtop
> Issue Type: Improvement
> Components: blueprints
> Affects Versions: backlog
> Reporter: RJ Nowling
> Assignee: RJ Nowling
> Priority: Minor
> Labels: bigpetstore
> Fix For: 0.9.0
>
> Attachments: BIGTOP-1366.patch
>
> Original Estimate: 8,736h
> Remaining Estimate: 8,736h
>
> BigPetStore uses synthetic data as the basis for its workflow. BPS's current
> model for generating customer data is sufficient for basic testing of the
> Hadoop ecosystem, **but the model is very basic and lacks sufficient
> complexity for embedding interesting patterns into the data**.
> As a result, **more complex, scalable testing such as testing clustering
> algorithms in Mahout on non-trivial data or multidimensional data with
> factors influencing it** is not currently possible.
> Efforts are currently underway to incrementally improve the current model
> (see BIGTOP-1271 and BIGTOP-1272).
> To create a model that can that incorporate **realistic, non-hierarchichal
> patterns** and input data to generate rich customer/transaction data with
> interesting correlations will require a re-imagining of the current model and
> its framework.
> To support the improvements to the model in BigPetStore, I have been working
> on an **alternative ab initio model, developed from scratch**. Since the
> development of a new model involves substantial R&D work with more
> specialized tools (mathematical and plotting libraries), I'm doing the
> current work outside of BPS using the iPython Notebook environment. Due to
> the long time frame, the model will be developed on a separate timeline to
> prevent slowing the development of BPS.
> Once the model has stabilized, I will begin incorporating the model into BPS
> itself. One option is to implement the model in using Scala for clean
> integration with **spark** which is likely to play an increasingly important
> role in the hadoop ecosystem, and thus will be an important part of
> bigpetstore as a test/blueprint app.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
