+1 (binding) On Wed, Nov 30, 2016 at 10:40 PM Henry Saputra <henry.sapu...@gmail.com> wrote:
> Hi All, > > As the champion for Griffin, I would like to start VOTE to bring the > project as Apache incubator podling. > > Here is the direct quote from the abstract: > > " > Griffin is a Data Quality Service platform built on Apache Hadoop and > Apache Spark. It provides a framework process for defining data > quality model, executing data quality measurement, automating data > profiling and validation, as well as a unified data quality > visualization across multiple data systems. It tries to address the > data quality challenges in big data and streaming context. > " > > Please cast your vote: > > [ ] +1, bring Griffin into Incubator > [ ] +0, I don't care either way, > [ ] -1, do not bring Griffin into Incubator, because... > > This vote will be open at least for 72 hours and only votes from the > Incubator PMC are binding. > > The VOTE will end 12/5 9am PST to pass through weekend. > > > Here is the link to the proposal: > > https://wiki.apache.org/incubator/GriffinProposal > > I have copied the proposal below for easy access > > > Thanks, > > - Henry > > > > Griffin Proposal > > Abstract > > Griffin is a Data Quality Service platform built on Apache Hadoop and > Apache Spark. It provides a framework process for defining data > quality model, executing data quality measurement, automating data > profiling and validation, as well as a unified data quality > visualization across multiple data systems. It tries to address the > data quality challenges in big data and streaming context. > > Proposal > > Griffin is a open source Data Quality solution for distributed data > systems at any scale in both streaming or batch data context. When > people use open source products (e.g. Apache Hadoop, Apache Spark, > Apache Kafka, Apache Storm), they always need a data quality service > to build his/her confidence on data quality processed by those > platforms. Griffin creates a unified process to define and construct > data quality measurement pipeline across multiple data systems to > provide: > > Automatic quality validation of the data > Data profiling and anomaly detection > Data quality lineage from upstream to downstream data systems. > Data quality health monitoring visualization > Shared infrastructure resource management > > Overview of Griffin > > Griffin has been deployed in production at eBay serving major data > systems, it takes a platform approach to provide generic features to > solve common data quality validation pain points. Firstly, user can > register the data asset which user wants to do data quality check. The > data asset can be batch data in RDBMS (e.g.Teradata), Apache Hadoop > system or near real-time streaming data from Apache Kafka, Apache > Storm and other real time data platforms. Secondly, user can create > data quality model to define the data quality rule and metadata. > Thirdly, the model or rule will be executed automatically (by the > model engine) to get the sample data quality validation results in a > few seconds for streaming data. Finally, user can analyze the data > quality results through built-in visualization tool to take actions. > > Griffin includes: > > Data Quality Model Engine > > Griffin is model driven solution, user can choose various data quality > dimension to execute his/her data quality validation based on selected > target data-set or source data-set ( as the golden reference data). It > has a corresponding library supporting it in back-end for the > following measurement: > > Accuracy - Does data reflect the real-world objects or a verifiable source > Completeness - Is all necessary data present > Validity - Are all data values within the data domains specified by the > business > Timeliness - Is the data available at the time needed > Anomaly detection - Pre-built algorithm functions for the > identification of items, events or observations which do not conform > to an expected pattern or other items in a dataset > Data Profiling - Apply statistical analysis and assessment of data > values within a dataset for consistency, uniqueness and logic. > > Data Collection Layer > > We support two kinds of data sources, batch data and real time data. > > For batch mode, we can collect data source from Apache Hadoop based > platform by various data connectors. > > For real time mode, we can connect with messaging system like Kafka to > near real time analysis. > > Data Process and Storage Layer > > For batch analysis, our data quality model will compute data quality > metrics in our spark cluster based on data source in Apache Hadoop. > > For near real time analysis, we consume data from messaging system, > then our data quality model will compute our real time data quality > metrics in our spark cluster. for data storage, we use time series > database in our back end to fulfill front end request. > > Griffin Service > > We have RESTful web services to accomplish all the functionalities of > Griffin, such as register data asset, create data quality model, > publish metrics, retrieve metrics, add subscription, etc. So, the > developers can develop their own user interface based on these web > services. > > Background > > At eBay, when people play with big data in Apache Hadoop (or other > streaming data), data quality often becomes one big challenge. > Different teams have built customized data quality tools to detect and > analyze data quality issues within their own domain. We are thinking > to take a platform approach to provide shared Infrastructure and > generic features to solve common data quality pain points. This would > enable us to build trusted data assets. > > Currently it’s very difficult and costly to do data quality validation > when we have big data flow across multi-platforms at eBay (e.g. > Oracle, Apache Hadoop, Couchbase, Apache Cassandra, Apache Kafka, > MongoDB). Take eBay real time personalization platform as an example. > Every day we have to validate data quality status for ~600M records ( > imagine we have 150M active users for our website). Data quality often > becomes one big challenge both in its streaming and batch pipelines. > > So we conclude 3 data quality problems at eBay: > > Lack of end2end unified view of data quality measurement from multiple > data sources to target applications, it usually takes a long time to > identify and fix poor data quality. > How to get data quality measured in streaming mode, we need to have a > process and tool to visualize data quality insights through > registering dataset which you want to check data quality, creating > data quality measurement model, executing the data quality validation > job and getting metrics insights for action taking. > No Shared platform and API Service, have to apply and manage own > hardware and software infrastructure. > > Rationale > > The challenge we face at eBay is that our data volume is becoming > bigger and bigger, system processes become more complex, while we do > not have a unified data quality solution to ensure the trusted data > sets which provide confidences on data quality to our data consumers. > The key challenges on data quality includes: > > Existing commercial data quality solution cannot address data quality > lineage among systems, cannot scale out to support fast growing data > at eBay > Existing eBay's domain specific tools take a long time to identify and > fix poor data quality when data flowed through multiple systems > Business logic becomes complex, requires data quality system much flexible. > > Some data quality issues do have business impact on user experiences, > revenue, efficiency & compliance. > > Communication overhead of data quality metrics, typically in a big > organization, which involve different teams. > > The idea of Griffin is to provide Data Quality validation as a > Service, to allow data engineers and data consumers to have: > > Near real-time understanding of the data quality health of your data > pipelines with end-to-end monitoring, all in one place. > Profiling, detecting and correlating issues and providing > recommendations that drive rapid and focused troubleshooting > A centralized data quality model management system including rule, > metadata, scheduler etc. > Native code generation to run everywhere, including Hadoop, Kafka, Spark, > etc. > One set of tools to build data quality pipelines across all eBay data > platforms. > > Current Status > > Meritocracy > > Griffin has been deployed in production at eBay and provided the > centralized data quality service for several eBay systems ( for > example, real time personalization platform, eBay real time ID linking > platform, Hadoop datasets, Site speed analytics platform). Our aim is > to build a diverse developer and user community following the Apache > meritocracy model. We will encourage contributions and participation > of all types of work, and ensure that contributors are appropriately > recognized. > > Community > > Currently the project is being developed at eBay. It's only for eBay > internal community. Griffin seeks to develop the developer and user > communities during incubation. We believe it will grow substantially > by becoming an Apache project. > > Core Developers > > Griffin is currently being designed and developed by engineers from > eBay Inc. – William Guo, Alex Lv, Shawn Sha, Vincent Zhao, John Liu. > All of these core developers have deep expertise in Apache Hadoop and > the Hadoop Ecosystem in general. > > Alignment > > The ASF is a natural host for Griffin given that it is already the > home of Hadoop, Beam, HBase, Hive, Storm, Kafka, Spark and other > emerging big data products. Those are requiring data quality solution > by nature to ensure the data quality which they processed. When people > use open source data technology, the big question to them is that how > we can ensure the data quality in it. Griffin leverages lot of Apache > open-source products. Griffin was designed to enable real time > insights into data quality validation by shared Infrastructure and > generic features to solve common data quality pain points. > > Known Risks > > Orphaned Products > > The core developers of Griffin team work full time on this project. > There is no risk of Griffin getting orphaned since at least one large > company (eBay) is extensively using it in their production Hadoop and > Spark clusters for multiple data systems. For example, currently there > are 4 data systems at eBay (real time personalization platform, eBay > real time ID linking platform, Hadoop, Site speed analytics platform) > are leveraging Griffin, with more than ~600M records for data quality > status validation every day, 35 data sets being monitored, 50+ data > quality models have been created. > > As Griffin is designed to connect many types of data sources, we are > very confident that they will use Griffin as a service for ensuring > the data quality in open source data ecosystems. We plan to extend and > diversify this community further through Apache. > > Inexperience with Open Source > > Griffin's core engineers are all active users and followers of open > source projects. They are already committers and contributors to the > Griffin Github project. All have been involved with the source code > that has been released under an open source license, and several of > them also have experience developing code in an open source > environment. Though the core set of Developers do not have Apache Open > Source experience, there are plans to onboard individuals with Apache > open source experience on to the project. > > Homogenous Developers > > The core developers are from eBay. Apache Incubation process > encourages an open and diverse meritocratic community. Griffin intends > to make every possible effort to build a diverse, vibrant and involved > community. We are committed to recruiting additional committers from > other companies based on their contribution to the project. > > Reliance on Salaried Developers > > eBay invested in Griffin as a company-wide data quality service > platform and some of its key engineers are working full time on the > project. they are all paid by eBay. We look forward to other Apache > developers and researchers to contribute to the project. > > Relationships with Other Apache Products > > Griffin has a strong relationship and dependency with Apache Hadoop, > Apache HBase, Apache Spark, Apache Kafka and Apache Storm, Apache > Hive. In addition, since there is a growing need for data quality > solution for open source platform (e.g. Hadoop, Kafka, Spark etc), > being part of Apache’s Incubation community, could help with a closer > collaboration among these four projects and as well as others. > > Documentation > > Information about Griffin can be found at https://github.com/eBay/griffin > > Initial Source > > Griffin has been under development since early 2016 by a team of > engineers at eBay Inc. It is currently hosted on Github.com under an > Apache license 2.0 at https://github.com/eBay/griffin . Once in > incubation we will be moving the code base to apache git library. > > External Dependencies > > Griffin has the following external dependencies. > > Basic > > JDK 1.7+ > Scala > Apache Maven > JUnit > Log4j > Slf4j > Apache Commons > > Hadoop > > Apache Hadoop > Apache HBase > Apache Hive > > DB > > InfluxData > > Apache Spark > > Spark Core Library > > REST Service > > Jersey > Spring MVC > > Web frontend > > AngularJS > jQuery > Bootstrap > RequireJS > eCharts > Font Awesome > > Cryptography > > Currently there's no cryptography in Griffin. > > Required Resources > > Mailing List > > We currently use eBay mail box to communicate, but we'd like to move > that to ASF maintained mailing lists. > > Current mailing list: ebay-griffin-d...@googlegroups.com > > Proposed ASF maintained lists: > > priv...@griffin.incubator.apache.org > > d...@griffin.incubator.apache.org > > comm...@griffin.incubator.apache.org > > Subversion Directory > > Git is the preferred source control system. > > Issue Tracking > > JIRA > > Other Resources > > The existing code already has unit tests so we will make use of > existing Apache continuous testing infrastructure. The resulting load > should not be very large. > > Initial Committers > > William Go > Alex Lv > Vincent Zhao > Shawn Sha > John Liu > Liang Shao > > Affiliations > > The initial committers are employees of eBay Inc. > > Sponsors > > Champion > > Henry Saputra (hsapu...@apache.org) > > Nominated Mentors > > Kasper Sørensen (kasper...@apache.org) > > Uma Maheswara Rao Gangumalla (umamah...@apache.org) > > Luciano Resende (luckbr1...@gmail.com) > > Sponsoring Entity > > We are requesting the Incubator to sponsor this project. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > > -- Sent from my Mobile device
