+1 (binding)

2016-12-01 17:58 GMT-08:00 Julian Hyde <jh...@apache.org>:

> +1 (binding)
>
> > On Dec 1, 2016, at 3:30 PM, Liang Chen <chenliang6...@gmail.com> wrote:
> >
> > Hi
> >
> > +1(non-binding)
> >
> > Regards
> > Liang
> >
> > Henry Saputra wrote
> >> Hi All,
> >>
> >> As the champion for Griffin, I would like to start VOTE to bring  the
> >> project as Apache incubator podling.
> >>
> >> Here is the direct quote from the abstract:
> >>
> >> "
> >> Griffin is a Data Quality Service platform built on Apache Hadoop and
> >> Apache Spark. It provides a framework process for defining data
> >> quality model, executing data quality measurement, automating data
> >> profiling and validation, as well as a unified data quality
> >> visualization across multiple data systems. It tries to address the
> >> data quality challenges in big data and streaming context.
> >> "
> >>
> >> Please cast your vote:
> >>
> >> [ ] +1, bring Griffin into Incubator
> >> [ ] +0, I don't care either way,
> >> [ ] -1, do not bring Griffin into Incubator, because...
> >>
> >> This vote will be open at least for 72 hours and only votes from the
> >> Incubator PMC are binding.
> >>
> >> The VOTE will end 12/5 9am PST to pass through weekend.
> >>
> >>
> >> Here is the link to the proposal:
> >>
> >> https://wiki.apache.org/incubator/GriffinProposal
> >>
> >> I have copied the proposal below for easy access
> >>
> >>
> >> Thanks,
> >>
> >> - Henry
> >>
> >>
> >>
> >> Griffin Proposal
> >>
> >> Abstract
> >>
> >> Griffin is a Data Quality Service platform built on Apache Hadoop and
> >> Apache Spark. It provides a framework process for defining data
> >> quality model, executing data quality measurement, automating data
> >> profiling and validation, as well as a unified data quality
> >> visualization across multiple data systems. It tries to address the
> >> data quality challenges in big data and streaming context.
> >>
> >> Proposal
> >>
> >> Griffin is a open source Data Quality solution for distributed data
> >> systems at any scale in both streaming or batch data context. When
> >> people use open source products (e.g. Apache Hadoop, Apache Spark,
> >> Apache Kafka, Apache Storm), they always need a data quality service
> >> to build his/her confidence on data quality processed by those
> >> platforms. Griffin creates a unified process to define and construct
> >> data quality measurement pipeline across multiple data systems to
> >> provide:
> >>
> >> Automatic quality validation of the data
> >> Data profiling and anomaly detection
> >> Data quality lineage from upstream to downstream data systems.
> >> Data quality health monitoring visualization
> >> Shared infrastructure resource management
> >>
> >> Overview of Griffin
> >>
> >> Griffin has been deployed in production at eBay serving major data
> >> systems, it takes a platform approach to provide generic features to
> >> solve common data quality validation pain points. Firstly, user can
> >> register the data asset which user wants to do data quality check. The
> >> data asset can be batch data in RDBMS (e.g.Teradata), Apache Hadoop
> >> system or near real-time streaming data from Apache Kafka, Apache
> >> Storm and other real time data platforms. Secondly, user can create
> >> data quality model to define the data quality rule and metadata.
> >> Thirdly, the model or rule will be executed automatically (by the
> >> model engine) to get the sample data quality validation results in a
> >> few seconds for streaming data. Finally, user can analyze the data
> >> quality results through built-in visualization tool to take actions.
> >>
> >> Griffin includes:
> >>
> >> Data Quality Model Engine
> >>
> >> Griffin is model driven solution, user can choose various data quality
> >> dimension to execute his/her data quality validation based on selected
> >> target data-set or source data-set ( as the golden reference data). It
> >> has a corresponding library supporting it in back-end for the
> >> following measurement:
> >>
> >> Accuracy - Does data reflect the real-world objects or a verifiable
> source
> >> Completeness - Is all necessary data present
> >> Validity - Are all data values within the data domains specified by the
> >> business
> >> Timeliness - Is the data available at the time needed
> >> Anomaly detection - Pre-built algorithm functions for the
> >> identification of items, events or observations which do not conform
> >> to an expected pattern or other items in a dataset
> >> Data Profiling - Apply statistical analysis and assessment of data
> >> values within a dataset for consistency, uniqueness and logic.
> >>
> >> Data Collection Layer
> >>
> >> We support two kinds of data sources, batch data and real time data.
> >>
> >> For batch mode, we can collect data source from Apache Hadoop based
> >> platform by various data connectors.
> >>
> >> For real time mode, we can connect with messaging system like Kafka to
> >> near real time analysis.
> >>
> >> Data Process and Storage Layer
> >>
> >> For batch analysis, our data quality model will compute data quality
> >> metrics in our spark cluster based on data source in Apache Hadoop.
> >>
> >> For near real time analysis, we consume data from messaging system,
> >> then our data quality model will compute our real time data quality
> >> metrics in our spark cluster. for data storage, we use time series
> >> database in our back end to fulfill front end request.
> >>
> >> Griffin Service
> >>
> >> We have RESTful web services to accomplish all the functionalities of
> >> Griffin, such as register data asset, create data quality model,
> >> publish metrics, retrieve metrics, add subscription, etc. So, the
> >> developers can develop their own user interface based on these web
> >> services.
> >>
> >> Background
> >>
> >> At eBay, when people play with big data in Apache Hadoop (or other
> >> streaming data), data quality often becomes one big challenge.
> >> Different teams have built customized data quality tools to detect and
> >> analyze data quality issues within their own domain. We are thinking
> >> to take a platform approach to provide shared Infrastructure and
> >> generic features to solve common data quality pain points. This would
> >> enable us to build trusted data assets.
> >>
> >> Currently it’s very difficult and costly to do data quality validation
> >> when we have big data flow across multi-platforms at eBay (e.g.
> >> Oracle, Apache Hadoop, Couchbase, Apache Cassandra, Apache Kafka,
> >> MongoDB). Take eBay real time personalization platform as an example.
> >> Every day we have to validate data quality status for ~600M records (
> >> imagine we have 150M active users for our website). Data quality often
> >> becomes one big challenge both in its streaming and batch pipelines.
> >>
> >> So we conclude 3 data quality problems at eBay:
> >>
> >> Lack of end2end unified view of data quality measurement from multiple
> >> data sources to target applications, it usually takes a long time to
> >> identify and fix poor data quality.
> >> How to get data quality measured in streaming mode, we need to have a
> >> process and tool to visualize data quality insights through
> >> registering dataset which you want to check data quality, creating
> >> data quality measurement model, executing the data quality validation
> >> job and getting metrics insights for action taking.
> >> No Shared platform and API Service, have to apply and manage own
> >> hardware and software infrastructure.
> >>
> >> Rationale
> >>
> >> The challenge we face at eBay is that our data volume is becoming
> >> bigger and bigger, system processes become more complex, while we do
> >> not have a unified data quality solution to ensure the trusted data
> >> sets which provide confidences on data quality to our data consumers.
> >> The key challenges on data quality includes:
> >>
> >> Existing commercial data quality solution cannot address data quality
> >> lineage among systems, cannot scale out to support fast growing data
> >> at eBay
> >> Existing eBay's domain specific tools take a long time to identify and
> >> fix poor data quality when data flowed through multiple systems
> >> Business logic becomes complex, requires data quality system much
> >> flexible.
> >>
> >> Some data quality issues do have business impact on user experiences,
> >> revenue, efficiency & compliance.
> >>
> >> Communication overhead of data quality metrics, typically in a big
> >> organization, which involve different teams.
> >>
> >> The idea of Griffin is to provide Data Quality validation as a
> >> Service, to allow data engineers and data consumers to have:
> >>
> >> Near real-time understanding of the data quality health of your data
> >> pipelines with end-to-end monitoring, all in one place.
> >> Profiling, detecting and correlating issues and providing
> >> recommendations that drive rapid and focused troubleshooting
> >> A centralized data quality model management system including rule,
> >> metadata, scheduler etc.
> >> Native code generation to run everywhere, including Hadoop, Kafka,
> Spark,
> >> etc.
> >> One set of tools to build data quality pipelines across all eBay data
> >> platforms.
> >>
> >> Current Status
> >>
> >> Meritocracy
> >>
> >> Griffin has been deployed in production at eBay and provided the
> >> centralized data quality service for several eBay systems ( for
> >> example, real time personalization platform, eBay real time ID linking
> >> platform, Hadoop datasets, Site speed analytics platform). Our aim is
> >> to build a diverse developer and user community following the Apache
> >> meritocracy model. We will encourage contributions and participation
> >> of all types of work, and ensure that contributors are appropriately
> >> recognized.
> >>
> >> Community
> >>
> >> Currently the project is being developed at eBay. It's only for eBay
> >> internal community. Griffin seeks to develop the developer and user
> >> communities during incubation. We believe it will grow substantially
> >> by becoming an Apache project.
> >>
> >> Core Developers
> >>
> >> Griffin is currently being designed and developed by engineers from
> >> eBay Inc. – William Guo, Alex Lv, Shawn Sha, Vincent Zhao, John Liu.
> >> All of these core developers have deep expertise in Apache Hadoop and
> >> the Hadoop Ecosystem in general.
> >>
> >> Alignment
> >>
> >> The ASF is a natural host for Griffin given that it is already the
> >> home of Hadoop, Beam, HBase, Hive, Storm, Kafka, Spark and other
> >> emerging big data products. Those are requiring data quality solution
> >> by nature to ensure the data quality which they processed. When people
> >> use open source data technology, the big question to them is that how
> >> we can ensure the data quality in it. Griffin leverages lot of Apache
> >> open-source products. Griffin was designed to enable real time
> >> insights into data quality validation by shared Infrastructure and
> >> generic features to solve common data quality pain points.
> >>
> >> Known Risks
> >>
> >> Orphaned Products
> >>
> >> The core developers of Griffin team work full time on this project.
> >> There is no risk of Griffin getting orphaned since at least one large
> >> company (eBay) is extensively using it in their production Hadoop and
> >> Spark clusters for multiple data systems. For example, currently there
> >> are 4 data systems at eBay (real time personalization platform, eBay
> >> real time ID linking platform, Hadoop, Site speed analytics platform)
> >> are leveraging Griffin, with more than ~600M records for data quality
> >> status validation every day, 35 data sets being monitored, 50+ data
> >> quality models have been created.
> >>
> >> As Griffin is designed to connect many types of data sources, we are
> >> very confident that they will use Griffin as a service for ensuring
> >> the data quality in open source data ecosystems. We plan to extend and
> >> diversify this community further through Apache.
> >>
> >> Inexperience with Open Source
> >>
> >> Griffin's core engineers are all active users and followers of open
> >> source projects. They are already committers and contributors to the
> >> Griffin Github project. All have been involved with the source code
> >> that has been released under an open source license, and several of
> >> them also have experience developing code in an open source
> >> environment. Though the core set of Developers do not have Apache Open
> >> Source experience, there are plans to onboard individuals with Apache
> >> open source experience on to the project.
> >>
> >> Homogenous Developers
> >>
> >> The core developers are from eBay. Apache Incubation process
> >> encourages an open and diverse meritocratic community. Griffin intends
> >> to make every possible effort to build a diverse, vibrant and involved
> >> community. We are committed to recruiting additional committers from
> >> other companies based on their contribution to the project.
> >>
> >> Reliance on Salaried Developers
> >>
> >> eBay invested in Griffin as a company-wide data quality service
> >> platform and some of its key engineers are working full time on the
> >> project. they are all paid by eBay. We look forward to other Apache
> >> developers and researchers to contribute to the project.
> >>
> >> Relationships with Other Apache Products
> >>
> >> Griffin has a strong relationship and dependency with Apache Hadoop,
> >> Apache HBase, Apache Spark, Apache Kafka and Apache Storm, Apache
> >> Hive. In addition, since there is a growing need for data quality
> >> solution for open source platform (e.g. Hadoop, Kafka, Spark etc),
> >> being part of Apache’s Incubation community, could help with a closer
> >> collaboration among these four projects and as well as others.
> >>
> >> Documentation
> >>
> >> Information about Griffin can be found at https://github.com/eBay/
> griffin
> >>
> >> Initial Source
> >>
> >> Griffin has been under development since early 2016 by a team of
> >> engineers at eBay Inc. It is currently hosted on Github.com under an
> >> Apache license 2.0 at https://github.com/eBay/griffin . Once in
> >> incubation we will be moving the code base to apache git library.
> >>
> >> External Dependencies
> >>
> >> Griffin has the following external dependencies.
> >>
> >> Basic
> >>
> >> JDK 1.7+
> >> Scala
> >> Apache Maven
> >> JUnit
> >> Log4j
> >> Slf4j
> >> Apache Commons
> >>
> >> Hadoop
> >>
> >> Apache Hadoop
> >> Apache HBase
> >> Apache Hive
> >>
> >> DB
> >>
> >> InfluxData
> >>
> >> Apache Spark
> >>
> >> Spark Core Library
> >>
> >> REST Service
> >>
> >> Jersey
> >> Spring MVC
> >>
> >> Web frontend
> >>
> >> AngularJS
> >> jQuery
> >> Bootstrap
> >> RequireJS
> >> eCharts
> >> Font Awesome
> >>
> >> Cryptography
> >>
> >> Currently there's no cryptography in Griffin.
> >>
> >> Required Resources
> >>
> >> Mailing List
> >>
> >> We currently use eBay mail box to communicate, but we'd like to move
> >> that to ASF maintained mailing lists.
> >>
> >> Current mailing list:
> >
> >> ebay-griffin-devs@
> >
> >>
> >> Proposed ASF maintained lists:
> >
> >> private@.apache
> >
> >>
> >
> >> dev@.apache
> >
> >>
> >
> >> commits@.apache
> >
> >>
> >> Subversion Directory
> >>
> >> Git is the preferred source control system.
> >>
> >> Issue Tracking
> >>
> >> JIRA
> >>
> >> Other Resources
> >>
> >> The existing code already has unit tests so we will make use of
> >> existing Apache continuous testing infrastructure. The resulting load
> >> should not be very large.
> >>
> >> Initial Committers
> >>
> >> William Go
> >> Alex Lv
> >> Vincent Zhao
> >> Shawn Sha
> >> John Liu
> >> Liang Shao
> >>
> >> Affiliations
> >>
> >> The initial committers are employees of eBay Inc.
> >>
> >> Sponsors
> >>
> >> Champion
> >>
> >> Henry Saputra (
> >
> >> hsaputra@
> >
> >> )
> >>
> >> Nominated Mentors
> >>
> >> Kasper Sørensen (
> >
> >> kaspersor@
> >
> >> )
> >>
> >> Uma Maheswara Rao Gangumalla (
> >
> >> umamahesh@
> >
> >> )
> >>
> >> Luciano Resende (
> >
> >> luckbr1975@
> >
> >> )
> >>
> >> Sponsoring Entity
> >>
> >> We are requesting the Incubator to sponsor this project.
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail:
> >
> >> general-unsubscribe@.apache
> >
> >> For additional commands, e-mail:
> >
> >> general-help@.apache
> >
> >
> >
> >
> >
> > --
> > View this message in context: http://apache-incubator-
> general.996316.n3.nabble.com/VOTE-Bring-Griffin-to-Apache-
> Incubator-tp52753p52763.html <http://apache-incubator-
> general.996316.n3.nabble.com/VOTE-Bring-Griffin-to-Apache-
> Incubator-tp52753p52763.html>
> > Sent from the Apache Incubator - General mailing list archive at
> Nabble.com <http://nabble.com/>.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> <mailto:general-unsubscr...@incubator.apache.org>
> > For additional commands, e-mail: general-h...@incubator.apache.org
> <mailto:general-h...@incubator.apache.org>
>

Reply via email to