Thanks, Jay. I would be in favor of trying to get what you have into Bigtop so that it can develop as part of this community. I'm excited to contribute to it and see it mature. You mentioned doing it as a separate module (i.e. a peer to bigtop-deploy and bigtop-packages) and I think that's the right approach. It's only adding new functionality, so there's no risk of breaking existing things and it's good to commit this sooner rather than later so it can develop quickly. This is a really cool way to demonstrate and even test Bigtop.
On Wed, Apr 9, 2014 at 1:16 PM, jay vyas (JIRA) <[email protected]> wrote: > > [ > https://issues.apache.org/jira/browse/BIGTOP-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13964562#comment-13964562] > > jay vyas commented on BIGTOP-1089: > ---------------------------------- > > FYI, as per some conversations here at ApacheCon, Im going to fast track a > patch for this so that folks such as [~mackrorysd] and [~posix4e] can > jump in and contribute (thanks in advance) ! > > FYI the current places for improvement (will create JIRAs) > > - rewrite pom.xml into a build.gradle script (how does gradle support a > notion of "profiles")? > - test hive code at scale and add configurability for jdbc connection > (right now is just local/embedded) > - test mahout code at scale > - update docs for above ^^ > - package BPS web app into a jetty app that launches as a build profile or > task in code (right now its in gh-pages branch on github) > > > > BigPetStore: A polyglot big data processing blueprint inside of bigtop > for comparing and learning about the tools in the bigtop packaged hadoop > ecosystem. > > > ---------------------------------------------------------------------------------------------------------------------------------------------------------- > > > > Key: BIGTOP-1089 > > URL: https://issues.apache.org/jira/browse/BIGTOP-1089 > > Project: Bigtop > > Issue Type: New Feature > > Components: Blueprints > > Affects Versions: 0.7.0 > > Reporter: jay vyas > > Assignee: jay vyas > > Fix For: 0.8.0 > > > > > > The need for templates for processing big data pipelines is obvious - > and also - given the increasing amount of overlap across different big data > and nosql projects, it will provide a ground truth in the future for > comparing the behaviour and approach of different tools to solve a common, > easily comprehended problem. > > This ticket formalizes the conversation in mailing list archives > regarding the BigPetStore proposal. > > At the moment, (with the exception of word count), there are very few > examples of bigdata problems that have been solved by a variety of > different technologies. And, even with wordcount, there arent alot of > templates which can be customized for applications. > > Comparatively: Other application developer communities (i.e.the Rails > folks, those using maven archetypes, etc.. ) have a plethora of template > applications which can be used to kickstart their applications and use > cases. > > This big pet store JIRA thus aims to do the following: > > 0) Curate a single, central, standard input data set . (modified: > generating a large input data set on the fly). > > 1) Define a big data processing pipeline (using the pet store theme - > except morphing it to be analytics rather than transaction oriented), and > implement basic aggregations in hive, pig, etc... > > 2) Sink the results of 2 into some kind of NoSQL store or search engine. > > > > Some implementation details -- open to change these, please > comment/review -- . > > - initial data source will be raw text or (better yet) some kind of > automatically generated data. > > - the source will initially go in bigtop/blueprints > > - the application sources can be in any modern JVM language > (java,scala,groovy,clojure), since bigtop supports scala, java, groovy > natively already and clojure is easy to support with the right jars. > > - each "job" will be named according to the corresponding DAG of the big > data pipeline . > > - all jobs should (not sure if requirement?) be controlled by a global > program (maybe oozie?) which runs the tasks in order, and can easily be > customized to use different tools at different stages. > > - for now, all outputs will be to files: so that users don't require > servers to run the app. > > - final data sinks will be into a highly available transaction oriented > store (solr/hbase/...) > > This ticket will be completed once a first iteration of BigPetStore is > complete using 3 ecosystem components, along with a depiction of the > pipeline which can be used for development. > > I've assigned this to myself :) I hope thats okay? Seems like at the > moment im the only one working on it. > > > > -- > This message was sent by Atlassian JIRA > (v6.2#6252) >
