Repository: incubator-griffin Updated Branches: refs/heads/master f0a43db1c -> f07926411
2018 07 19 doc update 1 Author: William Guo <[email protected]> Closes #360 from guoyuepeng/2018_07_19_doc_update_1. Project: http://git-wip-us.apache.org/repos/asf/incubator-griffin/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-griffin/commit/f0792641 Tree: http://git-wip-us.apache.org/repos/asf/incubator-griffin/tree/f0792641 Diff: http://git-wip-us.apache.org/repos/asf/incubator-griffin/diff/f0792641 Branch: refs/heads/master Commit: f079264116a9197ea4a77827abf314e2a7957bc4 Parents: f0a43db Author: William Guo <[email protected]> Authored: Thu Jul 19 18:49:32 2018 +0800 Committer: William Guo <[email protected]> Committed: Thu Jul 19 18:49:32 2018 +0800 ---------------------------------------------------------------------- griffin-doc/FSD.md | 162 --------------- griffin-doc/img/Business_Process.png | Bin 33482 -> 0 bytes griffin-doc/img/bark-sm.gif | Bin 2746 -> 0 bytes griffin-doc/img/data quality Flow.png | Bin 74065 -> 0 bytes griffin-doc/img/fsd/image2018-1-8 12-27-53.png | Bin 180417 -> 0 bytes griffin-doc/img/fsd/image2018-1-8 12-33-2.png | Bin 128529 -> 0 bytes griffin-doc/img/fsd/image2018-1-8 12-38-12.png | Bin 123654 -> 0 bytes griffin-doc/img/fsd/image2018-1-8 12-44-30.png | Bin 169651 -> 0 bytes griffin-doc/img/fsd/image2018-1-8 12-48-20.png | Bin 137816 -> 0 bytes griffin-doc/img/fsd/image2018-1-8 12-51-13.png | Bin 150127 -> 0 bytes griffin-doc/img/fsd/image2018-1-8 13-02-15.png | Bin 238736 -> 0 bytes griffin-doc/img/fsd/image2018-1-8 13-07-16.png | Bin 128242 -> 0 bytes griffin-doc/img/fsd/image2018-1-8 13-10-28.png | Bin 142898 -> 0 bytes griffin-doc/img/fsd/image2018-1-8 13-12-10.png | Bin 113922 -> 0 bytes griffin-doc/img/fsd/image2018-1-8 13-13-40.png | Bin 139110 -> 0 bytes griffin-doc/img/fsd/image2018-1-8 13-15-30.png | Bin 133758 -> 0 bytes griffin-doc/img/fsd/image2018-2-5 14-44-20.png | Bin 210289 -> 0 bytes griffin-doc/img/fsd/image2018-2-5 14-45-10.png | Bin 127324 -> 0 bytes griffin-doc/img/fsd/image2018-2-5 14-46-26.png | Bin 118602 -> 0 bytes griffin-doc/img/fsd/image2018-2-5 14-50-10.png | Bin 72787 -> 0 bytes griffin-doc/img/fsd/image2018-2-6 10-10-15.png | Bin 82679 -> 0 bytes griffin-doc/img/fsd/image2018-2-6 10-48-10.png | Bin 80211 -> 0 bytes griffin-doc/img/fsd/image2018-2-6 10-50-10.png | Bin 100417 -> 0 bytes griffin-doc/img/fsd/image2018-2-6 10-51-16.png | Bin 170904 -> 0 bytes griffin-doc/img/fsd/image2018-2-6 10-54-15.png | Bin 139184 -> 0 bytes griffin-doc/img/fsd/image2018-2-6 10-57-20.png | Bin 264629 -> 0 bytes griffin-doc/img/fsd/image2018-2-6 10-58-24.png | Bin 92768 -> 0 bytes griffin-doc/img/fsd/image2018-2-6 11-1-10.png | Bin 213911 -> 0 bytes griffin-doc/img/fsd/image2018-2-6 11-3-19.png | Bin 97685 -> 0 bytes griffin-doc/img/tdd/arch_design.png | Bin 110699 -> 0 bytes griffin-doc/img/tdd/class_diagram.png | Bin 26041 -> 0 bytes griffin-doc/img/tdd/model_design.png | Bin 51817 -> 0 bytes griffin-doc/img/tdd/mvc.png | Bin 22710 -> 0 bytes griffin-doc/img/techstack.png | Bin 127993 -> 0 bytes griffin-doc/intro.md | 8 - griffin-doc/proposal.md | 217 -------------------- griffin-doc/tdd.md | 83 -------- griffin-doc/ui/test-case.md | 62 ------ 38 files changed, 532 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/FSD.md ---------------------------------------------------------------------- diff --git a/griffin-doc/FSD.md b/griffin-doc/FSD.md deleted file mode 100644 index 4634ff8..0000000 --- a/griffin-doc/FSD.md +++ /dev/null @@ -1,162 +0,0 @@ - - -<!-- -Licensed to the Apache Software Foundation (ASF) under one -or more contributor license agreements. See the NOTICE file -distributed with this work for additional information -regarding copyright ownership. The ASF licenses this file -to you under the Apache License, Version 2.0 (the -"License"); you may not use this file except in compliance -with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, -software distributed under the License is distributed on an -"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -KIND, either express or implied. See the License for the -specific language governing permissions and limitations -under the License. ---> -# Functional Specification Document - -## Goals - -Apache Griffin is a model driven open source Data Quality solution for distributed data systems at any scale in both streaming or batch data context. When people use open source products (e.g. Hadoop, Spark, Kafka, Storm), they always need a data quality service to build his/her confidence on data quality processed by those platforms. Apache Griffin creates a unified process to define and construct data quality measurement pipeline across multiple data systems to provide: - -- Automatic quality validation of the data -- Data profiling and anomaly detection -- Data quality lineage from upstream to downstream data systems. -- Data quality health monitoring visualization -- Shared infrastructure resource management - -## Background - -When people play with big data in Hadoop (or other streaming data), data quality often becomes one big challenge. Different teams have built customized data quality tools to detect and analyze data quality issues within their own domain. We are thinking to take a platform approach to provide shared Infrastructure and generic features to solve common data quality pain points. This would enable us to build trusted data assets. - -Currently itâs very difficult and costly to do data quality validation when we have big data flow across multi-platforms (e.g. Oracle, Hadoop, Couchbase, Cassandra, Kafka, Mongodb). Take eBay real time personalization platform as a sample, everyday we have to validate data quality status for ~600M records ( imaging we have 150M active users for our website). Data quality often becomes one big challenge both in its streaming and batch pipelines. - -So we conclude 3 data quality problems at eBay: - -1. Lack of end2end unified view of data quality measurement from multiple data sources to target applications, it usually takes a long time to identify and fix poor data quality. -2. How to get data quality measured in streaming mode, we need to have a process and tool to visualize data quality insights through registering dataset which you want to check data quality, creating data quality measurement model, executing the data quality validation job and getting metrics insights for action taking. -3. No Shared platform and API Service, have to apply and manage own hardware and software infrastructure. - -## Assumptions - -- Users will primarily access this application from a PC -- We are handling textual data, no binary or encoded data - -## Main business process - - - -## Feature List - -<table class="confluenceTable" border="2"><tbody><tr><th class="confluenceTh">#</th><th class="confluenceTh">Title</th><th class="confluenceTh">User Story</th><th class="confluenceTh">Importance</th><th colspan="1" class="confluenceTh">Status</th><th class="confluenceTh">Notes</th></tr><tr><td class="confluenceTd">1</td><td class="confluenceTd">User login</td><td class="confluenceTd"><p>User can login the system so that I'm able to access<br />the subsequent features.</p></td><td class="confluenceTd">Must Have</td><td colspan="1" class="confluenceTd"> </td><td class="confluenceTd"><ul><li>There are two login strategies you can choose</li><li>Ldap strategy: Use LDAP non-anonymous bind for user authentication</li><li>Default strategy: You can login with any username and password</li></ul></td></tr><tr><td class="confluenceTd">2</td><td class="confluenceTd">Create measure - Accuracy</td><td class="confluenceTd"><p>User can create an accuracy measure, so that the metrics can be calculat ed by the rule defined in the measure </p></td><td class="confluenceTd">Must Have</td><td colspan="1" class="confluenceTd"> </td><td class="confluenceTd"><ul><li>5 Steps in wizard:</li></ul><ol><li>Select the source dataset and fields which will be used for comparision</li><li>Select the target dataset and fields which will be used for comparision</li><li>Mapping the target fields with source, some basic operator should be supported when mapping: =, !=, <,> etc.</li><li>Set partition configuration for source dataset and target dataset</li><li>Set basic configuration for your measure: name, description <br />measure type, etc.</li></ol><ul><li>When submitting the form, need to confirm the form info</li><li>After submitting, measure list should be displayed and the new measure is <br />displayed as the 1st record.</li></ul></td></tr><tr><td colspan="1" class="confluenceTd">3</td><td colspan="1" class="confluenceTd"><span>Create measure -Data Profiling</span></td><td colspan="1" clas s="confluenceTd"><p><span>User can create a profiling measure, so that the metrics </span><span>can be </span><span>calculated by the rule defined in the measure </span></p></td><td colspan="1" class="confluenceTd">Must Have</td><td colspan="1" class="confluenceTd"> </td><td colspan="1" class="confluenceTd"><ul><li>4 steps in wizard:</li></ul><ol><li>Select the target dataset and field which want to be checked, only one field should be selected.</li><li>Define your syntax check logic which will be applied on the selected field,<br />including: <strong>Null Count,Distinct Count, Total Count, Maximum, </strong><br /><strong>Minimum, </strong><strong>Average, Enum Detection Top5 Count</strong></li><li>Set partition configuration for target dataset</li><li><span>Set basic configuration for your measure: name,description, </span><br /><span>etc.</span><span><br /></span></li></ol><ul><li>When submitting the form, need to confirm the form info</li><li>After submitting, measure list should be displayed and the new measure is <br />displayed as the 1st record.</li></ul></td></tr><tr><td colspan="1" class="confluenceTd">4</td><td colspan="1" class="confluenceTd">List measures</td><td colspan="1" class="confluenceTd"><p>User can get a list of all measures which contain basic measure definition info</p></td><td colspan="1" class="confluenceTd"><span>Must Have</span></td><td colspan="1" class="confluenceTd"> </td><td colspan="1" class="confluenceTd"><ul><li>Information should be shown: Measure Name,Measure Type, Description, Action</li></ul></td></tr><tr><td colspan="1" class="confluenceTd">5</td><td colspan="1" class="confluenceTd">View a measure</td><td colspan="1" class="confluenceTd">User can view the details of a measure definition</td><td colspan="1" class="confluenceTd"><span>Must Have</span></td><td colspan="1" class="confluenceTd"> </td><td colspan="1" class="confluenceTd"><ul><li>All the information should be displayed: measure name, description, measure type,<b r />source,target,owner,etc.</li><li>Mapping rules and accuracy calculations formula are also displayed</li></ul></td></tr><tr><td colspan="1" class="confluenceTd">6</td><td colspan="1" class="confluenceTd">Delete a measure</td><td colspan="1" class="confluenceTd">User can delete my own measures</td><td colspan="1" class="confluenceTd">May Have</td><td colspan="1" class="confluenceTd"> </td><td colspan="1" class="confluenceTd"><ul><li>Administrator should also have delete permission</li></ul></td></tr><tr><td colspan="1" class="confluenceTd">7</td><td colspan="1" class="confluenceTd"><p><span>Create Job</span></p></td><td colspan="1" class="confluenceTd"><p><span>User can create a job which schedules one measure you created </span></p></td><td colspan="1" class="confluenceTd"><span>Must Have</span></td><td colspan="1" class="confluenceTd"> </td><td colspan="1" class="confluenceTd"><ul><li>1 step in wizard:</li></ul><ol><li><span>Set basic information for your job: measure name, cron expression, </span><span>data range, etc.</span></li></ol><ul><li>When submitting the form, need to confirm the form info</li><li>After submitting, job list should be displayed and the new job is <br />displayed as the 1st record.</li></ul></td></tr><tr><td colspan="1" class="confluenceTd">8</td><td colspan="1" class="confluenceTd"><p>List jobs</p></td><td colspan="1" class="confluenceTd"><p>User can get a list of all jobs definition info</p><td colspan="1" class="confluenceTd"><span>Must Have</span></td><td colspan="1" class="confluenceTd"> </td><td colspan="1" class="confluenceTd"><ul><li>Job information should be shown: Job Name,Previous Fire Time,Next Fire Time, Trigger State,Cron Expression, Action,Metric</li><li>Job instance information should be shown:AppID,Time,State</li><li>Jobinstance sorted by Time" desc</li></ul></td></tr><tr><td colspan="1" class="confluenceTd">9</td><td colspan="1" class="confluenceTd">Delete a job</td><td colspan="1" class="confluenceTd">User ca n delete my own jobs</td><td colspan="1" class="confluenceTd">May Have</td><td colspan="1" class="confluenceTd"> </td><td colspan="1" class="confluenceTd"><ul><li>Administrator should also have delete permission</li></ul></td></tr><tr><td colspan="1" class="confluenceTd">10</td><td colspan="1" class="confluenceTd">DataAsset list</td><td colspan="1" class="confluenceTd">User can view all data assets</td><td colspan="1" class="confluenceTd">Must Have</td><td colspan="1" class="confluenceTd"> </td><td colspan="1" class="confluenceTd"><ul><li>Information shown: table name, DB name, owner, creation time, location</li></ul></td></tr><tr><td colspan="1" class="confluenceTd">11</td><td colspan="1" class="confluenceTd">Heatmap</td><td colspan="1" class="confluenceTd"><p>User can see the heatmap when I login, so that I'll know</p><p>the status of the metrics</p></td><td colspan="1" class="confluenceTd"><span>Must Have</span></td><td colspan="1" class="confluenceTd"> </td><td colspan="1" class ="confluenceTd"><ul><li>Show all metrics results displayed as green</li></ul></td></tr><tr><td colspan="1" class="confluenceTd">12</td><td colspan="1" class="confluenceTd">Siderbar</td><td colspan="1" class="confluenceTd">The application can provide the list of all metrics</td><td colspan="1" class="confluenceTd">Must Have</td><td colspan="1" class="confluenceTd"> </td><td colspan="1" class="confluenceTd"><ul><li>List format: time, name, value.</li><li>Can be organized by Measure->metrics list</li></ul></td></tr><tr><td colspan="1" class="confluenceTd">13</td><td colspan="1" class="confluenceTd">Show mydashboard</td><td colspan="1" class="confluenceTd"><span>The application can display the charts of all metrics</span></td><td colspan="1" class="confluenceTd"><span>Must Have</span></td><td colspan="1" class="confluenceTd"> </td><td colspan="1" class="confluenceTd"><ul><li>A list of charts orgnized by <span>Measure->metrics list</span></li><li><span>Can filter by Measure</span>< /li></ul></td></tr><tr><td colspan="1" class="confluenceTd">14</td><td colspan="1" class="confluenceTd">Show metric detail</td><td colspan="1" class="confluenceTd"><span>Can display the detail of the metric</span></td><td colspan="1" class="confluenceTd"><span>Must Have</span></td><td colspan="1" class="confluenceTd"> </td><td colspan="1" class="confluenceTd"><ul><li><span>Show metric details</span></li></ul></td></tr><tr><td colspan="1" class="confluenceTd">15</td><td colspan="1" class="confluenceTd">Scheduler job</td><td colspan="1" class="confluenceTd"><p>The application should have the scheduler jobs to</p><p>calculate the metrics</p></td><td colspan="1" class="confluenceTd">Must Have</td><td colspan="1" class="confluenceTd"> </td><td colspan="1" class="confluenceTd"><ul><li>According to the rules defined in the measures, the scheduler job has these<br />functions:</li></ul><ol><li>Know when to calculate a metric</li><li>Call the measure engine to get metrics values</li><li>Save the metrics result</li></ol></td></tr><tr><td colspan="1" class="confluenceTd">16</td><td colspan="1" class="confluenceTd">Measure engine</td><td colspan="1" class="confluenceTd">The application can calculate the metrics values</td><td colspan="1" class="confluenceTd">Must Have</td><td colspan="1" class="confluenceTd"> </td><td colspan="1" class="confluenceTd"><ul><li>With the rule definition, the engine can calculate the metrics values</li></ul></td></tr></tbody></table> - -## User interaction and design - -<p style="color: rgb(51,102,255);">#1 User Login</p> - -<img src="img/fsd/image2018-1-8 12-27-53.png" > - -<p style="color: rgb(51,102,255);">#2 Create measure - Accuracy</p> -step 1 : Select the source dataset and fields which will be used for comparision - -<img src="img/fsd/image2018-1-8 12-33-2.png" > - -step 2 : Select the target dataset and fields which will be used for comparision - -<img src="img/fsd/image2018-1-8 12-38-12.png" > - -step 3 : Mapping the target fields with source - -<img src="img/fsd/image2018-1-8 12-44-30.png" > - -step 4 : Set partition configuration for source dataset and target dataset - -<img src="img/fsd/image2018-1-8 12-48-20.png" > - -step 5 : Set basic configuration for your measure - -<img src="img/fsd/image2018-1-8 12-51-13.png" > - -confirmation : - -<img src="img/fsd/image2018-1-8 13-02-15.png" > - -<p style="color: rgb(51,102,255);">#3 Create measure - Profiling</p> -step 1 : Select the target dataset and field which want to be checked - -<img src="img/fsd/image2018-1-8 13-07-16.png" > - -step 2 : Define your syntax check logic which will be applied on the selected field - -<img src="img/fsd/image2018-1-8 13-10-28.png" > - -step 3 : Set partition configuration for target dataset - -<img src="img/fsd/image2018-1-8 13-12-10.png" > - -step 4 : Set basic configuration for your measure - -<img src="img/fsd/image2018-1-8 13-13-40.png" > - -confirmation : - -<img src="img/fsd/image2018-1-8 13-15-30.png" > - -<p style="color: rgb(51,102,255);">#4 List measures</p> -<img src="img/fsd/image2018-2-5 14-45-10.png" > - -<p style="color: rgb(51,102,255);">#5 View a measure</p> -<img src="img/fsd/image2018-2-5 14-44-20.png" > - -<p style="color: rgb(51,102,255);">#6 Delete a measure</p> -<img src="img/fsd/image2018-2-5 14-46-26.png" > - -<p style="color: rgb(51,102,255);">#7 Create job</p> - -<img src="img/fsd/image2018-2-6 11-1-10.png" > - -<img src="img/fsd/image2018-2-6 11-3-19.png" > - -<p style="color: rgb(51,102,255);">#8 List jobs</p> - -<img src="img/fsd/image2018-2-6 10-54-15.png" > - -<img src="img/fsd/image2018-2-6 10-57-20.png" > - -<p style="color: rgb(51,102,255);">#9 Delete a job</p> - -<img src="img/fsd/image2018-2-6 10-58-24.png" > - -<p style="color: rgb(51,102,255);">#10 DataAsset list</p> -<img src="img/fsd/image2018-2-5 14-50-10.png" > - -<p style="color: rgb(51,102,255);">#11 Heatmap</p> -<img src="img/fsd/image2018-2-6 10-10-15.png" > - -<p style="color: rgb(51,102,255);">#12 List all metrics values</p> -<img src="img/fsd/image2018-2-6 10-48-10.png" height="500px"> - - - -<p style="color: rgb(51,102,255);">#13 Show My Dashboard</p> -chart list -> Max chart display - -<img src="img/fsd/image2018-2-6 10-50-10.png" height="400px" > - -<p style="color: rgb(51,102,255);">#14 Show metric detail</p> -<img src="img/fsd/image2018-2-6 10-51-16.png" height="400px" > - -<p style="color: rgb(51,102,255);">#15 Scheduler job</p> -Same UI as #7 - -## Questions -Below is a list of questions to be addressed as a result of this requirements document: -<table border="2"><tbody><tr><th>Question</th><th>Outcome</th></tr></tbody></table> - -## Not Doing -- http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/Business_Process.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/Business_Process.png b/griffin-doc/img/Business_Process.png deleted file mode 100644 index ff0f25f..0000000 Binary files a/griffin-doc/img/Business_Process.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/bark-sm.gif ---------------------------------------------------------------------- diff --git a/griffin-doc/img/bark-sm.gif b/griffin-doc/img/bark-sm.gif deleted file mode 100644 index 04c9d09..0000000 Binary files a/griffin-doc/img/bark-sm.gif and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/data quality Flow.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/data quality Flow.png b/griffin-doc/img/data quality Flow.png deleted file mode 100644 index 79d5c8a..0000000 Binary files a/griffin-doc/img/data quality Flow.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-1-8 12-27-53.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-1-8 12-27-53.png b/griffin-doc/img/fsd/image2018-1-8 12-27-53.png deleted file mode 100644 index 810b0d9..0000000 Binary files a/griffin-doc/img/fsd/image2018-1-8 12-27-53.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-1-8 12-33-2.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-1-8 12-33-2.png b/griffin-doc/img/fsd/image2018-1-8 12-33-2.png deleted file mode 100644 index 0347a07..0000000 Binary files a/griffin-doc/img/fsd/image2018-1-8 12-33-2.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-1-8 12-38-12.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-1-8 12-38-12.png b/griffin-doc/img/fsd/image2018-1-8 12-38-12.png deleted file mode 100644 index e627f11..0000000 Binary files a/griffin-doc/img/fsd/image2018-1-8 12-38-12.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-1-8 12-44-30.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-1-8 12-44-30.png b/griffin-doc/img/fsd/image2018-1-8 12-44-30.png deleted file mode 100644 index db799c5..0000000 Binary files a/griffin-doc/img/fsd/image2018-1-8 12-44-30.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-1-8 12-48-20.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-1-8 12-48-20.png b/griffin-doc/img/fsd/image2018-1-8 12-48-20.png deleted file mode 100644 index 9c621ef..0000000 Binary files a/griffin-doc/img/fsd/image2018-1-8 12-48-20.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-1-8 12-51-13.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-1-8 12-51-13.png b/griffin-doc/img/fsd/image2018-1-8 12-51-13.png deleted file mode 100644 index c3a8a6e..0000000 Binary files a/griffin-doc/img/fsd/image2018-1-8 12-51-13.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-1-8 13-02-15.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-1-8 13-02-15.png b/griffin-doc/img/fsd/image2018-1-8 13-02-15.png deleted file mode 100644 index 39ca2fa..0000000 Binary files a/griffin-doc/img/fsd/image2018-1-8 13-02-15.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-1-8 13-07-16.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-1-8 13-07-16.png b/griffin-doc/img/fsd/image2018-1-8 13-07-16.png deleted file mode 100644 index b38dfe8..0000000 Binary files a/griffin-doc/img/fsd/image2018-1-8 13-07-16.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-1-8 13-10-28.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-1-8 13-10-28.png b/griffin-doc/img/fsd/image2018-1-8 13-10-28.png deleted file mode 100644 index 4715db9..0000000 Binary files a/griffin-doc/img/fsd/image2018-1-8 13-10-28.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-1-8 13-12-10.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-1-8 13-12-10.png b/griffin-doc/img/fsd/image2018-1-8 13-12-10.png deleted file mode 100644 index 1e6a4ae..0000000 Binary files a/griffin-doc/img/fsd/image2018-1-8 13-12-10.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-1-8 13-13-40.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-1-8 13-13-40.png b/griffin-doc/img/fsd/image2018-1-8 13-13-40.png deleted file mode 100644 index 522e2c1..0000000 Binary files a/griffin-doc/img/fsd/image2018-1-8 13-13-40.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-1-8 13-15-30.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-1-8 13-15-30.png b/griffin-doc/img/fsd/image2018-1-8 13-15-30.png deleted file mode 100644 index 97cd6e0..0000000 Binary files a/griffin-doc/img/fsd/image2018-1-8 13-15-30.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-2-5 14-44-20.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-2-5 14-44-20.png b/griffin-doc/img/fsd/image2018-2-5 14-44-20.png deleted file mode 100644 index 9789f4b..0000000 Binary files a/griffin-doc/img/fsd/image2018-2-5 14-44-20.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-2-5 14-45-10.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-2-5 14-45-10.png b/griffin-doc/img/fsd/image2018-2-5 14-45-10.png deleted file mode 100644 index c1f882a..0000000 Binary files a/griffin-doc/img/fsd/image2018-2-5 14-45-10.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-2-5 14-46-26.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-2-5 14-46-26.png b/griffin-doc/img/fsd/image2018-2-5 14-46-26.png deleted file mode 100644 index 2077e2f..0000000 Binary files a/griffin-doc/img/fsd/image2018-2-5 14-46-26.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-2-5 14-50-10.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-2-5 14-50-10.png b/griffin-doc/img/fsd/image2018-2-5 14-50-10.png deleted file mode 100644 index 70bba7e..0000000 Binary files a/griffin-doc/img/fsd/image2018-2-5 14-50-10.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-2-6 10-10-15.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-2-6 10-10-15.png b/griffin-doc/img/fsd/image2018-2-6 10-10-15.png deleted file mode 100644 index 12189fc..0000000 Binary files a/griffin-doc/img/fsd/image2018-2-6 10-10-15.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-2-6 10-48-10.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-2-6 10-48-10.png b/griffin-doc/img/fsd/image2018-2-6 10-48-10.png deleted file mode 100644 index a698e27..0000000 Binary files a/griffin-doc/img/fsd/image2018-2-6 10-48-10.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-2-6 10-50-10.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-2-6 10-50-10.png b/griffin-doc/img/fsd/image2018-2-6 10-50-10.png deleted file mode 100644 index 36a06e1..0000000 Binary files a/griffin-doc/img/fsd/image2018-2-6 10-50-10.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-2-6 10-51-16.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-2-6 10-51-16.png b/griffin-doc/img/fsd/image2018-2-6 10-51-16.png deleted file mode 100644 index aa796b6..0000000 Binary files a/griffin-doc/img/fsd/image2018-2-6 10-51-16.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-2-6 10-54-15.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-2-6 10-54-15.png b/griffin-doc/img/fsd/image2018-2-6 10-54-15.png deleted file mode 100644 index 0407309..0000000 Binary files a/griffin-doc/img/fsd/image2018-2-6 10-54-15.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-2-6 10-57-20.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-2-6 10-57-20.png b/griffin-doc/img/fsd/image2018-2-6 10-57-20.png deleted file mode 100644 index eafa6fa..0000000 Binary files a/griffin-doc/img/fsd/image2018-2-6 10-57-20.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-2-6 10-58-24.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-2-6 10-58-24.png b/griffin-doc/img/fsd/image2018-2-6 10-58-24.png deleted file mode 100644 index 1f79d9b..0000000 Binary files a/griffin-doc/img/fsd/image2018-2-6 10-58-24.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-2-6 11-1-10.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-2-6 11-1-10.png b/griffin-doc/img/fsd/image2018-2-6 11-1-10.png deleted file mode 100644 index 7d3b75c..0000000 Binary files a/griffin-doc/img/fsd/image2018-2-6 11-1-10.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/fsd/image2018-2-6 11-3-19.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/fsd/image2018-2-6 11-3-19.png b/griffin-doc/img/fsd/image2018-2-6 11-3-19.png deleted file mode 100644 index dd094b8..0000000 Binary files a/griffin-doc/img/fsd/image2018-2-6 11-3-19.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/tdd/arch_design.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/tdd/arch_design.png b/griffin-doc/img/tdd/arch_design.png deleted file mode 100644 index 6170830..0000000 Binary files a/griffin-doc/img/tdd/arch_design.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/tdd/class_diagram.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/tdd/class_diagram.png b/griffin-doc/img/tdd/class_diagram.png deleted file mode 100644 index 6a6352e..0000000 Binary files a/griffin-doc/img/tdd/class_diagram.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/tdd/model_design.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/tdd/model_design.png b/griffin-doc/img/tdd/model_design.png deleted file mode 100644 index 2d18486..0000000 Binary files a/griffin-doc/img/tdd/model_design.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/tdd/mvc.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/tdd/mvc.png b/griffin-doc/img/tdd/mvc.png deleted file mode 100644 index 381da82..0000000 Binary files a/griffin-doc/img/tdd/mvc.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/img/techstack.png ---------------------------------------------------------------------- diff --git a/griffin-doc/img/techstack.png b/griffin-doc/img/techstack.png deleted file mode 100644 index ebc5540..0000000 Binary files a/griffin-doc/img/techstack.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/intro.md ---------------------------------------------------------------------- diff --git a/griffin-doc/intro.md b/griffin-doc/intro.md index 4558e10..25e6789 100644 --- a/griffin-doc/intro.md +++ b/griffin-doc/intro.md @@ -62,18 +62,10 @@ For near real time analysis, we consume data from messaging system, then our dat We have RESTful web services to accomplish all the functionalities of Apache Griffin, such as exploring data-sets, create data quality measures, publish metrics, retrieve metrics, add subscription, etc. So, the developers can develop their own user interface based on these web serivces. -## Main business process - - - ## Main architecture diagram  -## Main tech stack diagram - - - ## Rationale The challenge we face at big data ecosystem is that our data volume is becoming bigger and bigger, systems process become more complex, while we do not have a unified data quality solution to ensure the trusted data sets which provide confidences on data quality to our data consumers. The key challenges on data quality includes: http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/proposal.md ---------------------------------------------------------------------- diff --git a/griffin-doc/proposal.md b/griffin-doc/proposal.md deleted file mode 100644 index 16d9628..0000000 --- a/griffin-doc/proposal.md +++ /dev/null @@ -1,217 +0,0 @@ -<!-- -Licensed to the Apache Software Foundation (ASF) under one -or more contributor license agreements. See the NOTICE file -distributed with this work for additional information -regarding copyright ownership. The ASF licenses this file -to you under the Apache License, Version 2.0 (the -"License"); you may not use this file except in compliance -with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, -software distributed under the License is distributed on an -"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -KIND, either express or implied. See the License for the -specific language governing permissions and limitations -under the License. ---> - -## Abstract -Apache Griffin is a Data Quality Service platform built on Apache Hadoop and Apache Spark. It provides a framework process for defining data quality measure, executing data quality measurement, automating data profiling and validation, as well as a unified data quality visualization across multiple data systems. It tries to address the data quality challenges in big data and streaming context. - -## Proposal -Apache Griffin is a open source Data Quality solution for distributed data systems at any scale in both streaming or batch data context. When people use open source products (e.g. Apache Hadoop, Apache Spark, Apache Kafka, Apache Storm), they always need a data quality service to build his/her confidence on data quality processed by those platforms. Apache Griffin creates a unified process to define and construct data quality measurement pipeline across multiple data systems to provide: - - - Automatic quality validation of the data - - Data profiling and anomaly detection - - Data quality lineage from upstream to downstream data systems. - - Data quality health monitoring visualization - - Shared infrastructure resource management - -## Overview of Apache Griffin -Apache Griffin has been deployed in production at eBay serving major data systems, it takes a platform approach to provide generic features to solve common data quality validation pain points. Firstly, user can register the data-set which user wants to do data quality check. The data-set can be batch data in RDBMS (e.g.Teradata), Apache Hadoop system or near real-time streaming data from Apache Kafka, Apache Storm and other real time data platforms. Secondly, user can create data quality measure to define the data quality rule and metadata. Thirdly, the measure or rule will be executed automatically(by the measure engine) to get the sample data quality validation results in a few seconds for streaming data. Finally, user can analyze the data quality results through built-in visualization tool to take actions. - -Apache Griffin includes: - -**Data Quality Measure Engine**: Apache Griffin is model driven solution, user can choose various data quality dimension to execute his/her data quality validation based on selected target data-set or source data-set ( as the golden reference data). It has corresponding library supporting it in back-end for the following measurement: - - - Accuracy - Does data reflect the real-world objects or a verifiable source - - Completeness - Is all necessary data present - - Validity - Are all data values within the data domains specified by the business - - Timeliness - Is the data available at the time needed - - Anomaly detection - Pre-built algorithm functions for the identification of items, events or observations which do not conform to an expected pattern or other items in a dataset - - Data Profiling - Apply statistical analysis and assessment of data values within a dataset for consistency, uniqueness and logic. - -**Data Collection Layer**: - -We support two kinds of data sources, batch data and real time data. - -For batch mode, we can collect data source from our Hadoop platform by various data connectors. - -For real time mode, we can connect with messaging system like Kafka to near real time analysis. - -**Data Process and Storage Layer**: - -For batch analysis, our data quality measure will compute data quality metrics in our spark cluster based on data source in hadoop. - -For near real time analysis, we consume data from messaging system, then our data quality measure will compute our real time data quality metrics in our spark cluster. for data storage, we use time series database in our back end to fulfill front end request. - -**Apache Griffin Service**: - -We have RESTful web services to accomplish all the functionalities of Apache Griffin, such as register data-set, create data quality measure, publish metrics, retrieve metrics, add subscription, etc. So, the developers can develop their own user interface based on these web serivces. - -## Background -At eBay, when people play with big data in Apache Hadoop (or other streaming data), data quality often becomes one big challenge. Different teams have built customized data quality tools to detect and analyze data quality issues within their own domain. We are thinking to take a platform approach to provide shared Infrastructure and generic features to solve common data quality pain points. This would enable us to build trusted data assets. - -Currently itâs very difficult and costly to do data quality validation when we have big data flow across multi-platforms at eBay (e.g. Oracle, Apache Hadoop, Couchbase, Apache Cassandra, Apache Kafka, MongoDB). Take eBay real time personalization platform as a sample, everyday we have to validate data quality status for ~600M records ( imaging we have 150M active users for our website). Data quality often becomes one big challenge both in its streaming and batch pipelines. - -So we conclude 3 data quality problems at eBay: - -1. Lack of end2end unified view of data quality measurement from multiple data sources to target applications, it usually takes a long time to identify and fix poor data quality. -2. How to get data quality measured in streaming mode, we need to have a process and tool to visualize data quality insights through registering dataset which you want to check data quality, creating data quality measurement, executing the data quality validation job and getting metrics insights for action taking. -3. No Shared platform and API Service, have to apply and manage own hardware and software infrastructure. - -## Rationale -The challenge we face at eBay is that our data volume is becoming bigger and bigger, systems process become more complex, while we do not have a unified data quality solution to ensure the trusted data sets which provide confidences on data quality to our data consumers. The key challenges on data quality includes: - -1. Existing commercial data quality solution cannot address data quality lineage among systems, cannot scale out to support fast growing data at eBay -2. Existing eBay's domain specific tools take a long time to identify and fix poor data quality when data flowed through multiple systems -3. Business logic becomes complex, requires data quality system much flexible. -4. Some data quality issues do have business impact on user experiences, revenue, efficiency & compliance. -5. Communication overhead of data quality metrics, typically in a big organization, which involve different teams. - -The idea of Apache Griffin is to provide Data Quality validation as a Service, to allow data engineers and data consumers to have: - - - Near real-time understanding of the data quality health of your data pipelines with end-to-end monitoring, all in one place. - - Profiling, detecting and correlating issues and providing recommendations that drive rapid and focused troubleshooting - - A centralized data quality measure management system including rule, metadata, scheduler etc. - - Native code generation to run everywhere, including Hadoop, Kafka, Spark, etc. - - One set of tools to build data quality pipelines across all eBay data platforms. - -## Current Status -###Meritocracy -Apache Griffin has been deployed in production at eBay and provided the centralized data quality service for several eBay systems ( for example, real time personalization platform, eBay real time ID linking platform, Hadoop datasets, Site speed analytics platform). Our aim is to build a diverse developer and user community following the Apache meritocracy model. We will encourage contributions and participation of all types of work, and ensure that contributors are appropriately recognized. - -###Community -Currently the project is being developed at eBay. It's only for eBay internal community. Apache Griffin seeks to develop the developer and user communities during incubation. We believe it will grow substantially by becoming an Apache project. - -###Core Developers -Apache Griffin is currently being designed and developed by engineers from eBay Inc. â William Guo, Alex Lv, Shawn Sha, Vincent Zhao. All of these core developers have deep expertise in Apache Hadoop and the Hadoop Ecosystem in general. - -###Alignment -The ASF is a natural host for Apache Griffin given that it is already the home of Hadoop, Beam, HBase, Hive, Storm, Kafka, Spark and other emerging big data products. Those are requiring data quality solution by nature to ensure the data quality which they processed. When people use open source data technology, the big question to them is that how we can ensure the data quality in it. Apache Griffin leverages lot of Apache open-source products. Apache Griffin was designed to enable real time insights into data quality validation by shared Infrastructure and generic features to solve common data quality pain points. - -##Known Risks -###Orphaned Products -The core developers of Apache Griffin team work full time on this project. There is no risk of Apache Griffin getting orphaned since at least one large company (eBay) is extensively using it in their production Hadoop and Spark clusters for multiple data systems. For example, currently there are 4 data systems at eBay (real time personalization platform, eBay real time ID linking platform, Hadoop, Site speed analytics platform) are leveraging Apache Griffin, with more than ~600M records for data quality status validation every day, 35 data sets being monitored, 50+ data quality measures have been created. - -As Apache Griffin is designed to connect many types of data sources, we are very confident that they will use Apache Griffin as a service for ensuring the data quality in open source data ecosystems. We plan to extend and diversify this community further through Apache. - -###Inexperience with Open Source -Apache Griffin's core engineers are all active users and followers of open source projects. They are already committers and contributors to the Apache Griffin Github project. All have been involved with the source code that has been released under an open source license, and several of them also have experience developing code in an open source environment. Though the core set of Developers do not have Apache Open Source experience, there are plans to onboard individuals with Apache open source experience on to the project. - -###Homogenous Developers -The core developers are from eBay. Apache Incubation process encourages an open and diverse meritocratic community. Apache Griffin intends to make every possible effort to build a diverse, vibrant and involved community. We are committed to recruiting additional committers from other companies based on their contribution to the project. - -###Reliance on Salaried Developers -eBay invested in Apache Griffin as a company-wide data quality service platform and some of its key engineers are working full time on the project. they are all paid by eBay. We look forward to other Apache developers and researchers to contribute to the project. - -###Relationships with Other Apache Products -Apache Griffin has a strong relationship and dependency with Apache Hadoop, Apache HBase, Apache Spark, Apache Kafka and Apache Storm, Apache Hive. In addition, since there is a growing need for data quality solution for open source platform (e.g. Hadoop, Kafka, Spark etc), being part of Apacheâs Incubation community, could help with a closer collaboration among these four projects and as well as others. - -##Documentation -Information about Apache Griffin can be found at https://github.com/apache/incubator-griffin. - -##Initial Source -Apache Griffin has been under development since early 2016 by a team of engineers at eBay Inc. It is currently hosted on Github.com under an Apache license 2.0 at https://github.com/apache/incubator-griffin. - -Once in incubation we will be moving the code base to Apache git repository. - -##External Dependencies -Apache Griffin has the following external dependencies. - -**Basic** - -- JDK 1.7+ -- Scala -- Apache Maven -- JUnit -- Log4j -- Slf4j -- Apache Commons - -**Hadoop** - -- Apache Hadoop -- Apache HBase -- Apache Hive - -**DB** - -- MongoDB -- InfluxData - -**Apache Spark** - -- Spark Core Library - -**REST Service** - -- Jersey -- Spring MVC - -**Web frontend** - -- AngularJS -- jQuery -- Bootstrap -- RequireJS -- Font Awesome - -##Cryptography -Currently there's no cryptography in Apache Griffin. - -##Required Resources -###Mailing List### -We currently use eBay mail box to communicate, but we'd like to move that to ASF maintained mailing lists. - -Current mailing list: [email protected] - -Proposed ASF maintained lists: [email protected] - [email protected] - [email protected] - -###Subversion Directory -Git is the preferred source control system. - -###Issue Tracking -JIRA - -###Other Resources -The existing code already has unit tests so we will make use of existing Apache continuous testing infrastructure. The resulting load should not be very large. - -##Initial Committers -- William Guo <yueguo at ebay dot com> -- Alex Lv <lzhixing at ebay dot com> -- Vincent Zhao <wenzhao at ebay dot com> -- Shawn Sha <yosha at ebay dot com> -- Amber Vaidya <amvaidya at ebay dot com> -- Liang Shao <lshao at ebay dot com> - -##Affiliations -The initial committers are employees of eBay Inc. - -##Sponsors - - -##Champion -Henry Saputra <hsaputra at apache dot org> - Apache IPMC member - -##Nominated Mentors - -##Sponsoring Entity -We are requesting the Incubator to sponsor this project. http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/tdd.md ---------------------------------------------------------------------- diff --git a/griffin-doc/tdd.md b/griffin-doc/tdd.md deleted file mode 100644 index d2227b9..0000000 --- a/griffin-doc/tdd.md +++ /dev/null @@ -1,83 +0,0 @@ -<!-- -Licensed to the Apache Software Foundation (ASF) under one -or more contributor license agreements. See the NOTICE file -distributed with this work for additional information -regarding copyright ownership. The ASF licenses this file -to you under the Apache License, Version 2.0 (the -"License"); you may not use this file except in compliance -with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, -software distributed under the License is distributed on an -"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -KIND, either express or implied. See the License for the -specific language governing permissions and limitations -under the License. ---> -# Technical Design Document - - -## System Architecture -There are 3 main parts of the system: Web UI, Backend Services, and Measure engine on Hadoop. Please refer to the below high level architecture diagram - - - -## Web UI -- AngularJS 1.5.7 - https://angularjs.org/ -- eCharts 3.2.2 - http://echarts.baidu.com/ -- Bootstrap 3.3.6 - http://getbootstrap.com - -## Backend Service Layer -There are several key componets hosted on the web server, such as RESTful web service, Job scheduler, Core Service. We provides RESTful APIs to support all kinds of clients(web browser, mobile app, applications on PC, etc.). The Job scheduler is responsible for triggering the execution of the measures and persist the metrics values. The Metadata engine is to provide most of the management features such as measure management, data asset management, notification management, user settings, etc. - -### RESTful Service -#### Concepts -We follow the principles in https://codeplanet.io/principles-good-restful-api-design/ to build up our RESTful web services. - -**Root URL**: https://example.org/api/v1/* - -**Verbs**: -- GET (SELECT): Retrieve a specific Resource from the Server, or a listing of Resources. -- POST (CREATE): Create a new Resource on the Server. -- PUT (UPDATE): Update a Resource on the Server, providing the entire Resource. -- PATCH (UPDATE): Update a Resource on the Server, providing only changed attributes. -- DELETE (DELETE): Remove a Resource from the Server. - -#### Technical framework -Jersey: https://jersey.java.net/ - -And the resources should be authorized. Refer to https://jersey.java.net/documentation/latest/security.html - -### Job Scheduler -May leverage QUARTZ: http://www.quartz-scheduler.org/ - -<TBD> - -### Core Service -Here are the key components in Core Service: -- data asset management -- measure management -- metrics subscription -- notification component -- user settings - -For each component, we follow the MVC-based architecture. - - - -Here're the typical classes/interfaces for each component - - - -## Data Collection&Processing Layer - - -For data ingestion, griffin can consume real time data from streaming and also support batch data like sql or structured files. - -For data processing, griffin provides several measure libraries for various data quality dimensions like accuracy, validity, etc. - -After appending business rules into those measures, griffin will deploy related jobs in spark cluster. - -For more details of the design, please refer to https://github.com/eBay/DQSolution/blob/master/griffin-doc/measures.md http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/f0792641/griffin-doc/ui/test-case.md ---------------------------------------------------------------------- diff --git a/griffin-doc/ui/test-case.md b/griffin-doc/ui/test-case.md deleted file mode 100644 index 40ed3d7..0000000 --- a/griffin-doc/ui/test-case.md +++ /dev/null @@ -1,62 +0,0 @@ -<!-- -Licensed to the Apache Software Foundation (ASF) under one -or more contributor license agreements. See the NOTICE file -distributed with this work for additional information -regarding copyright ownership. The ASF licenses this file -to you under the Apache License, Version 2.0 (the -"License"); you may not use this file except in compliance -with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, -software distributed under the License is distributed on an -"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -KIND, either express or implied. See the License for the -specific language governing permissions and limitations -under the License. ---> -# Functional Test cases - - -| TestCase ID | Test Page | Test Case Description | Test Case Steps | Test Data | Expected Result | Actual Result | Test Result | Jira Bug ID | -| ----------- | -------------------------- | ---------------------------------------- | ---------------------------------------- | ---------------------------------------- | ---------------------------------------- | ------------- | ----------- | ----------- | -| 0101 | login page | invalid corp id - check user cannot login the system with invalid corp id. | 1. Input invalid corp id.<br>2. Input password.<br>3. click 'log in' button. | | 1. login failed. | | Passed | | -| 0102 | login page | invalid password - check user cannot login the system with invalid password. | 1. input valid corp id.<br>2.input invalid password<br>3.click 'log in' button. | | 1. login failed. | | Passed | | -| 0103 | login page | valid corp id and passoword - check user can login the system with valid corp id and password. | 1. Input the corp id and password.<br>2 click 'log in' button. | | 1. login succesfully | | Passed | | -| 0104 | login page | remember password | 1. Input the corp id and password.<br>2. select 'remember password'.<br>3.click 'log in' button.<br>4. close the brower.<br>5. open the brower again.<br>6. visit the griffin page. | | 1.the id and password are valid.<br>2.'remember password' is checked.<br>3.logged in the griffin homepage.<br>4.the brower is closed.<br>5.the brower is reopened.<br>6.the griffin homepage is opened, instead of the login page. | | Passed | | -| 0105 | login page | not remember password | 1. Input the corp id and password.<br>2. unselect 'remember password'.<br>3.click 'log in' button.<br>4. close the brower.<br>5. open the brower again.<br>6. visit the griffin page. | | 1.the id and password are valid.<br>2.'remember password' is unchecked.<br>3.logged in the griffin homepage.<br>4.the brower is closed.<br>5.the brower is reopened.<br>6.the login page is opened. | | Passed | | -| 0201 | main page | menu bar - check all links in the menu work. | 1. click 'Health'.<br>2. click 'Measures'.<br>3.click 'Create Measure'.<br>4.click 'Data Profiling'.<br>4. click your username -> 'API Docs'. | | 1.show 'health' page.<br>2.show 'measures' page.<br>3.show 'crreate measure' page<br>4.show 'data profiling' page<br>4.open new page for API page. | | Passed | | -| 0202 | main page | menu bar - search | 1.input a word in the search box.<br>2.do search. | | 1. show search result. | unimplemented | | | -| 0203 | main page | menu bar - user profile | 1. click username -> 'user profile' | | 1. show user profile page | unimplemented | | | -| 0204 | main page | menu bar - setting | 1. click username -> 'setting' | | 1. show setting page. | unimplemented | | | -| 0205 | main page | right side - DataAssets | 1. click 'DataAssets' link | | 1.show the data assets page. | | Passed | | -| 0206 | main page | right side - DQ Metrics | 1. click 'DQ Metrics' link. | | 1. show DQ Metrics page | | Passed | | -| 0207 | main page | right side - statistics for the DQ data. | 1. check the DQ data with the name, last updated time, and the data quality.<br>2. show more for one item, check the dq trend chart. <br>3. click the chart.<br>4. close the zoomed-in chart. | | 1.show all the dq data.<br>2.show the latest dq trend chart for the item.<br>3.the dq chart is zoomed in.<br>4.the zoomed-in chart is closed. | | Passed | | -| 0301 | health page | heatmap | 1. open 'heatmap' tab.<br>2. check the data quality metrics heatmap.<br>3. click inside the heatmap. | | 1.show the heatmap.<br>2.all the data are shown successfully.<br>3.show the metrics page. | | Passed | | -| 0302 | health page | check the UI layout when the page is zoomed in and out. | 1.zoom in the page.<br>2.zoom out the page. | | 1. display the page correctly. | | Passed | | -| 0401 | metrics page | check metrics data | 1. check the dq charts for the metrics.<br>2. click one chart. | | 1. all the data in the dq charts are correct.<br>2. the chart is zoomed in. | | Passed | | -| 0402 | metrics page | Download Sample | 1. click 'download sample'. | | 1. the sample is downloaded to the local path. | unimplemented | | | -| 0403 | metrics page | Profiling | 1. click 'profiling' | | 1. show 'profiling' | unimplemented | | | -| 0404 | metrics page | check the UI layout when the page is zoomed in and out. | 1.zoom in the page.<br>2.zoom out the page. | | 1. display the page correctly. | | Passed | | -| 0501 | Measures page | check the measures data | 1. check all the columns are correct or not.<br>2. click one measure name. | | 1. all the data are correct.<br>2. show more information of the measure. | | Passed | | -| 0502 | Measures page | delete measure | 1. click 'delete' icon for one measure.<br>2. confirm to delete the measure. | | 1. open delete confirmation page.<br>2. the measure is removed from the measures table. | | Passed | | -| 0503 | Measures page | View measure | 1. click 'view' icon for one measure. | | 1. open view measure page | | | | -| 0504 | Measures page | table paging | 1. click other pages in the measures table. | | 1.all the data in other pages are show correctly. | | Passed | | -| 0505 | Measures page | create measure | 1. click 'create measure' button | | 1. open 'create measure' page. | | Passed | | -| 0506 | Measures page | check the UI layout when the page is zoomed in and out. | 1.zoom in the page.<br>2.zoom out the page. | | 1. display the page correctly. | | Passed | | -| 0601 | create measure - accuracy | create accuracy | 1. click 'Measures' -> 'Create Measure' -> 'Accuracy'<br>2.choose the source. Select a schema, e.g. 'default -> demo_src'.<br>3.select one or more attributes, e.g. id, age.<br>4. click 'next'.<br>5. choose the target. Select a schema, e.g. 'default -> demo_tgt'.<br>6. select one or more attributes, e.g. id.<br>7. click 'next'.<br>8. select 'Map To',e.g.'='.<br>9. select a source field for each target.<br>10. click 'next'.<br>11. input the partition configuration for the source and the target,e.g. partition size'1day'.<br>12. input the required information, e.g. measure name 'test_acu'.<br>13.click 'submit'.<br>14. confirm to save. | source schema: 'default -> demo_src'.<br>Source attributes: id, age.<br>target schema: 'default -> demo_tgt'.<br>target attributes, e.g. id.<br>Map to: =.<br>Source Fields: default.demo_src.id.<br>Partition size: '1day'.<br>model name: 'test_acu'. <br> | 1. open 'crea te accuracy' page.<br>2. the source shcema is selected. The corresponding attributes are shown in the attributes table.<br>3. the source attributes are selected.<br>4. go to 'choose target' step.<br>5. the target schema is selected. The corresponding attributes are shown in the attributes table.<br>6. the target attributes are selected.<br>7. go to 'mapping source and target' step.<br>8. exactly map to the source.<br>9. the source field is selected for each target.<br>10. go to 'partition configuration' step.<br>12. the configuration are input correctly.<br>11. go to 'configuration' step.<br>12. the required info are input correctly.<br>13. open a confirmation page.<br>14. the new model 'test_acu' is created. It is shown in the measures table | | Passed | | -| 0602 | create measure - accuracy | show error message if no source attribute is selected. | 1. click 'Measures' -> 'Create Measure' -> 'Accuracy'.<br>2. click 'next' | | 1. open 'create accuracy' page.<br>2. show error message to select at least one attribute. | | Passed | | -| 0603 | create measure - accuracy | show error message if no target attribute is selected. | 1. click 'Measures' -> 'Create Measure' -> 'Accuracy'.<br>2.choose the source. Select a schema, e.g. 'default -> demo_src'.<br>3.select one or more attributes, e.g. id, age.<br>4. click 'next'.<br>5. in the 'target' step, click 'next'. | source schema: 'default -> demo_src'.<br>Source attributes: id, age. | "1. open 'create accuracy' page.<br>2. the source shcema is selected. The corresponding attributes are shown in the attributes table.<br>3. the source attributes are selected.<br>4. go to 'choose target' step.<br>5. show error message to select at least one attribute. | | Passed | | -| 0604 | create measure - accuracy | show error message if 'map fields' is not set. | 1. click 'Measures' -> 'Create Measure' -> 'Accuracy'<br>2.choose the source. Select a schema, e.g. 'default -> demo_src'.<br>3.select one or more attributes, e.g. id, age.<br>4. click 'next'.<br>5. choose the target. Select a schema, e.g. 'default -> demo_tgt'.<br>6. select one or more attributes, e.g. id.<br>7. click 'next'.<br>8. no selection. click 'next'.<br> | source schema: 'default -> demo_src'.<br>Source attributes: id, age.<br>target schema: 'default -> demo_tgt'.<br>target attributes, e.g. id.<br> | 1. open 'create accuracy' page.<br>2. the source shcema is selected. The corresponding attributes are shown in the attributes table.<br>3. the source attributes are selected.<br>4. go to 'choose target' step.<br>5. the target schema is selected. The corresponding attributes are shown in the attributes table.<br>6. the target attributes are selected.<br>7. go to 'mapping source and tar get' step.<br>8. no source field is selected for each target.<br>9. show error message. | | Passed | | -| 0605 | create measure - accuracy | show error if the configuration is invalid | 1. click 'Measures' -> 'Create Measure' -> 'Accuracy'<br>2.choose the source. Select a schema, e.g. 'default -> demo_src'.<br>3.select one or more attributes, e.g. id, age.<br>4. click 'next'.<br>5. choose the target. Select a schema, e.g. 'default -> demo_tgt'.<br>6. select one or more attributes, e.g. id.<br>7. click 'next'.<br>8. select 'Map To',e.g.'='.<br>9. select a source field for each target.<br>10. click 'next'.<br>11.input the partition configuration for the source and the target,e.g. partition size'1day'. <br>12. input invalid value for each field, e.g. measure name 'a test', | source schema: 'default -> demo_src'.<br>Source attributes: id, age.<br>target schema: 'default -> demo_tgt'.<br>target attributes, e.g. id.<br>Map to: =.<br>Source Fields: default.demo_src.id.<br>Partition size: '1day'.<br>model name: 'a test', <br> | 1. open 'create accuracy' page.<br>2. the source shcema i s selected. The corresponding attributes are shown in the attributes table.<br>3. the source attributes are selected.<br>4. go to 'choose target' step.<br>5. the target schema is selected. The corresponding attributes are shown in the attributes table.<br>6. the target attributes are selected.<br>7. go to 'mapping source and target' step.<br>8. the source field is selected for each target..<br>9. exactly map to the source.<br>10. the source field is selected for each target.<br>11. go to 'configuration' step.<br>12. show error for invalid value. | | Passed | | -| 0606 | create measure - accuracy | check the link to add new data asset. | 1. click the link for adding new data asset. | | 1. go to the 'register data asset' page. | unimplemented | | | -| 0607 | create measure - accuracy | check the UI layout for all the steps when the page is zoomed in and out. | 1.zoom in the page.<br>2.zoom out the page. | | 1.all the steps in the page can be shown correctly. | | Passed | | -| 0701 | create measure - profiling | check measure with profiling type can be created. | 1. click 'Measures' -> 'Create Measure' ->Data Profiling<br>2.choose the target. Select a schema, e.g. 'default -> demo_src'.<br>3.select one attribute, e.g. id.<br>4. click 'next'.<br>5. choose one profiling model, e.g. Null Count ,Total Count.<br>6. click 'next'.<br>7.input the partition configuration for the source,e.g. partition size'1day'. <br>8. click 'next'.<br>9. input valid value for the configuration fields. e.g. measure name 'avalidity'.<br>10. click 'submit'<br>11. click 'save'. | schema: 'default -> demo_src'.<br>Attribute: id.<br>validity model: Null Count ,Total Count.<br>measure name: 'a profiling', <br> | 1. open 'create profiling' page.<br>2. the target schem is selected. The corresponding attributes are shown in the attributes table.<br>3. the attribute is selected.<br>4. go to 'select models' page.<br>5. the profiling model is selected. The description of the model is shown as well.<br>7. go to 'partition configuration' step.<br>8. go to 'configuration' step.<br>9. all the value are valid.<br>10. open a confirmation page.<br>11. the new model 'profiling' is created successfully. it is shown in the measures page. | | Passed | | -| 0702 | create measure - profiling | show error if no target is selected. | 1. click 'Measures' -> 'Create Measure' ->Data Profiling<br>2. not choose the target.<br>3. click 'next'. | | 1. open 'create profiling' page.<br>2. no target schem is selected.<br>3. show error. | | Passed | | -| 0703 | create measure - profiling | show error if any field is invalid. | 1. click 'Measures' -> 'Create Measure' ->Data Profiling<br>2.choose the target. Select a schema, e.g. 'default -> demo_src'.<br>3.select one attribute, e.g. id.<br>4. click 'next'.<br>5. choose one profiling model, e.g. Null Count ,Total Count.<br>6. click 'next'.<br>7.input the partition configuration for the source,e.g. partition size'1day'. <br>8. click 'next'.<br>9. input invalid value for the configuration fields. | schema: 'default -> demo_src'.<br>Attribute: id.<br>validity model: Null Count ,Total Count.<br>measure name: 'a profiling', <br> | 1. open 'create profiling' page.<br>2. the target schem is selected. The corresponding attributes are shown in the attributes table.<br>3. the attribute is selected.<br>4. go to 'select models' page.<br>5. the profiling model is selected. The description of the model is shown as well.<br>6. go to 'partition configuration' step.<br>7. go to 'configur ation' step.<br>8. show error for the invalid value. | | Passed | | -| 0704 | create measure - profiling | check the UI layout for all the steps when the page is zoomed in and out. | 1.zoom in the page.<br>2.zoom out the page. | | 1.all the steps in the page can be shown correctly. | | Passed | | -| 0801 | data assets | check the data assets information | 1. check all the columns are correct or not.<br>2. show more for an asset. | | 1. all the data are correct.<br>2. show the schemas of the asset. | | Passed | | -| 0802 | data assets | table paging | 1. click other pages in the table. | | 1.all the data in other pages are show correctly. | | Passed | | -| 0803 | data assets | check the UI layout when the page is zoomed in and out. | 1.zoom in the page.<br>2.zoom out the page. | | 1. display the page correctly. | | Passed | | -
