Supporting Ceph will be very interesting. Could you please create a JIRA? Would be great if there is any patch.
On Thu, Jan 12, 2017 at 10:19 PM, Dmitry Buzolin <dmitry.buzo...@theice.com> wrote: > Also, besides supporting 3 different storage interfaces Ceph is more > sophisticated storage backend compare to Hadoop at this time. > For example: in addition to replicated pools, Ceph supports erasure coded > pools (kind of host based RAID), which has requires lot less storage > compare to the former. > Other great features of Ceph is an algorytmic approach to map data to the > nodes rather than having centrally managed namenodes and snapshots. I don't > think HDFS offers any of these features. In terms of performance, Ceph > should be faster than HFDS since it is written on C++ and because it > doesn't have scalability limitations when mapping data to storage pools, > compare to Hadoop, where name node is such point of contention. > > > -----Original Message----- > From: Paul Guo [mailto:paul...@gmail.com] > Sent: Wednesday, January 11, 2017 9:15 PM > To: dev@hawq.incubator.apache.org > Subject: Re: Different storage backends in HAWQ? > > WARNING - External email; exercise caution > > > HAWQ supports row oriented format also (AO Table). For block storage & > posix fs, I suspect you could use gpdb since gpdb uses local storage though > probably a bit additional work is needed. For object storage (e.g. S3 > interface), this is an interesting topic. There have been a JIRA for this > though there are some debates about better solutions. > > HAWQ-823 Amazon S3 External Table Support > https://issues.apache.org/jira/browse/HAWQ-823 > > > 2017-01-11 23:14 GMT+08:00 Dmitry Buzolin <dmitry.buzo...@theice.com>: > > > Since HAWQ only depends on Hadoop and Parquet as a columnar format. I > > would like to propose having Hawq having pluggable storage backends. > > Hadoop is already supported but there is Ceph storage backend which > offers > > standard Posix compliant file system, object and a block storage. Ceph is > > also is location aware and written in C++. I believe this is more > important > > than porting HAWQ to windows. Your thoughts? > > > > Thanks, > > Dmitry. > > > > ________________________________ > > > > This message may contain confidential information and is intended for > > specific recipients unless explicitly noted otherwise. If you have reason > > to believe you are not an intended recipient of this message, please > delete > > it and notify the sender. This message may not represent the opinion of > > Intercontinental Exchange, Inc. (ICE), its subsidiaries or affiliates, > and > > does not constitute a contract or guarantee. Unencrypted electronic mail > is > > not secure and the recipient of this message is expected to provide > > safeguards from viruses and pursue alternate means of communication where > > privacy or a binding message is desired. > > > > ________________________________ > > This message may contain confidential information and is intended for > specific recipients unless explicitly noted otherwise. If you have reason > to believe you are not an intended recipient of this message, please delete > it and notify the sender. This message may not represent the opinion of > Intercontinental Exchange, Inc. (ICE), its subsidiaries or affiliates, and > does not constitute a contract or guarantee. Unencrypted electronic mail is > not secure and the recipient of this message is expected to provide > safeguards from viruses and pursue alternate means of communication where > privacy or a binding message is desired. > -- Best Regards, Yandong
