Repository: kylin Updated Branches: refs/heads/document 88f1da47c -> 0752828a2
KYLIN-875 add document for arch design Project: http://git-wip-us.apache.org/repos/asf/kylin/repo Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/0752828a Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/0752828a Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/0752828a Branch: refs/heads/document Commit: 0752828a28133cadec698ffae6c50eddc0aed67e Parents: 88f1da4 Author: Yang Li <[email protected]> Authored: Sat Feb 13 20:19:14 2016 +0800 Committer: Yang Li <[email protected]> Committed: Sat Feb 13 20:19:14 2016 +0800 ---------------------------------------------------------------------- website/_data/development.yml | 1 + website/_dev/plugin_arch.md | 42 +++++++++++++++++++ .../develop/plugin_arch_adaptor_pattern.png | Bin 0 -> 50424 bytes .../develop/plugin_arch_factory_pattern.png | Bin 0 -> 62389 bytes website/images/develop/plugin_arch_overview.png | Bin 0 -> 151459 bytes 5 files changed, 43 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/kylin/blob/0752828a/website/_data/development.yml ---------------------------------------------------------------------- diff --git a/website/_data/development.yml b/website/_data/development.yml index 8b03f76..d3be519 100644 --- a/website/_data/development.yml +++ b/website/_data/development.yml @@ -25,6 +25,7 @@ - howto_docs - howto_package - howto_release + - plugin_arch - new_metadata - web_tech - about_temp_files http://git-wip-us.apache.org/repos/asf/kylin/blob/0752828a/website/_dev/plugin_arch.md ---------------------------------------------------------------------- diff --git a/website/_dev/plugin_arch.md b/website/_dev/plugin_arch.md new file mode 100644 index 0000000..35fc5e1 --- /dev/null +++ b/website/_dev/plugin_arch.md @@ -0,0 +1,42 @@ +--- +layout: dev +title: Plugin Architecture +categories: development +permalink: /development/plugin_arch.html +--- + +The plugin architecture aims to make Kylin extensible regarding computation framework, data source, and cube storage. As of v1, Kylin tightly couples with Hadoop MapReduce as computation framework, Hive as data source, and HBase as storage. Questions came like: could Kylin use Spark as cube engine, or how about a different storage like Cassandra. We want to be open to different options, and to make sure Kylin evolve with the best tech stacks. That is why the plugin architecture is introduced in Kylin v2. + + + +## How it Works + +The cube metadata defines the type of engine, source, and storage that a cube depends on. Factory pattern is used to construct instances of each dependency. Adaptor pattern is used to connect the parts together. + +For example a cube descriptor may contains: + +- fact_table: `SOME_HIVE_TABLE` +- engine_type: `2` (MR Engine v2) +- storage_type: `2` (HBase Storage v2) + +Based on the metadata, factories creates MR engine, Hive data source, and HBase storage. + + + +The engine is like a motherboard, on which source and storage must be plugged as defined by the IN and OUT interfaces. Data source and storage must adapt to the interfaces in order to be connected to engine motherboard. + + + +Once the above object graph is created and connected, engine can drive the cube build process. + +## The Benefits of Plugin Architecture + +- Freedom + - Zoo break, not bound to Hadoop any more + - Free to go to a better engine or storage +- Extensibility + - Accept any input, e.g. Kafka + - Embrace next-gen distributed platform, e.g. Spark +- Flexibility + - Choose different engine for different data set + http://git-wip-us.apache.org/repos/asf/kylin/blob/0752828a/website/images/develop/plugin_arch_adaptor_pattern.png ---------------------------------------------------------------------- diff --git a/website/images/develop/plugin_arch_adaptor_pattern.png b/website/images/develop/plugin_arch_adaptor_pattern.png new file mode 100644 index 0000000..af49430 Binary files /dev/null and b/website/images/develop/plugin_arch_adaptor_pattern.png differ http://git-wip-us.apache.org/repos/asf/kylin/blob/0752828a/website/images/develop/plugin_arch_factory_pattern.png ---------------------------------------------------------------------- diff --git a/website/images/develop/plugin_arch_factory_pattern.png b/website/images/develop/plugin_arch_factory_pattern.png new file mode 100644 index 0000000..473270b Binary files /dev/null and b/website/images/develop/plugin_arch_factory_pattern.png differ http://git-wip-us.apache.org/repos/asf/kylin/blob/0752828a/website/images/develop/plugin_arch_overview.png ---------------------------------------------------------------------- diff --git a/website/images/develop/plugin_arch_overview.png b/website/images/develop/plugin_arch_overview.png new file mode 100644 index 0000000..52af951 Binary files /dev/null and b/website/images/develop/plugin_arch_overview.png differ
