Love this discussion. Like to highlight 3 major roles HBase is playing
currently, so we don't miss any of them when looking for a replacement.
1) Storage: A high speed big data storage
2) Cache: A distributed storage cache layer (was BlockCache)
3) MPP: A distributed computation framework (was Coprocessor)
The "Storage" seems at the central of discussion. Be it Parquet, ORC, or a
new file format, to me the standard interface is most important. As long as
we have consensus on the access interface, like MapReduce / Spark Dataset,
then the rest of debate can be easily resolved by a fair benchmark. Also it
allows people with different preference to keep their own implementation
under the standard interface, and not impacting the rest of Kylin.
The "Cache" and the "MPP" were more or less overlooked. I suggest we pay
more attentions to them. Apart from Spark and Alluxio, any other
alternatives? Actually Druid is a well-rounded choice, as like HBase, it
covers all the 3 roles pretty well.
In general, I prefer to choose from the state of the art instead of
re-inventing. Indeed, Kylin is not a storage project. A new storage format
is not Kylin's mission. Any storage innovations we come across here would
be more beneficial if contribute to Parquet or ORC community.
Regards
Yang
On Tue, Oct 2, 2018 at 11:20 AM ShaoFeng Shi wrote:
> Hi Billy,
>
> Yes, the cloud storage should be considered. The traditional file layouts
> on HDFS may not work well on cloud storage. Kylin needs to allow extension
> here. I will add this to the requirement.
>
> Billy Liu 于2018年9月29日周六 下午3:22写道：
>
> > Hi Shaofeng,
> >
> > I'd like to add one more character: cloud-native storage support.
> > Quite a few users are using S3 on AWS, or Azure Data Lake Storage on
> > Azure. If new storage engine could be more cloud friendly, more user
> > could get benefits from it.
> >
> > With Warm regards
> >
> > Billy Liu
> > ShaoFeng Shi 于2018年9月28日周五 下午2:15写道：
> > >
> > > Hi Kylin developers.
> > >
> > > HBase has been Kylin’s storage engine since the first day; Kylin on
> HBase
> > > has been verified as a success which can support low latency & high
> > > concurrency queries on a very large data scale. Thanks to HBase, most
> > Kylin
> > > users can get on average less than 1-second query response.
> > >
> > > But we also see some limitations when putting Cubes into HBase; I
> shared
> > > some of them in the HBaseConf Asia 2018[1] this August. The typical
> > > limitations include:
> > >
> > >- Rowkey is the primary index, no secondary index so far;
> > >
> > > Filtering by row key’s prefix and suffix can get very different
> > performance
> > > result. So the user needs to do a good design about the row key;
> > otherwise,
> > > the query would be slow. This is difficult sometimes because the user
> > might
> > > not predict the filtering patterns ahead of cube design.
> > >
> > >- HBase is a key-value instead of a columnar storage
> > >
> > > Kylin combines multiple measures (columns) into fewer column families
> for
> > > smaller data size (row key size is remarkable). This causes HBase often
> > > needing to read more data than requested.
> > >
> > >- HBase couldn't run on YARN
> > >
> > > This makes the deployment and auto-scaling a little complicated,
> > especially
> > > in the cloud.
> > >
> > > In one word, HBase is complicated to be Kylin’s storage. The
> maintenance,
> > > debugging is also hard for normal developers. Now we’re planning to
> seek
> > a
> > > simple, light-weighted, read-only storage engine for Kylin. The new
> > > solution should have the following characteristics:
> > >
> > >- Columnar layout with compression for efficient I/O;
> > >- Index by each column for quick filtering and seeking;
> > >- MapReduce / Spark API for parallel processing;
> > >- HDFS compliant for scalability and availability;
> > >- Mature, stable and extensible;
> > >
> > > With the plugin architecture[2] introduced in Kylin 1.5, adding
> multiple
> > > storages to Kylin is possible. Some companies like Kyligence Inc and
> > > Meituan.com, have developed their customized storage engine for Kylin
> in
> > > their product or platform. In their experience, columnar storage is a
> > good
> > > supplement for the HBase engine. Kaisen Kang from Meituan.com has
> shared
> > > their KOD (Kylin on Druid) solution[3] in this August’s Kylin meetup in
> > > Beijing.
> > >
> > > We plan to do a PoC with Apache Parquet + Apache Spark in the next
> phase.
> > > Parquet is a standard columnar file format and has been widely
> supported
> > by
> > > many projects like Hive, Impala, Drill, etc. Parquet is adding the page
> > > level column index to support fine-grained filtering. Apache Spark can
> > > provide the parallel computing over Parquet and can be deployed on
> > > YARN/Mesos and Kubernetes. With this combination, the data persistence
> > and
> > > computation are separated, which makes the scaling in/out much easier
