[
https://issues.apache.org/jira/browse/S2GRAPH-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15232029#comment-15232029
]
Hyunsung Jo commented on S2GRAPH-1:
-----------------------------------
The [Redis data model document |
https://docs.google.com/presentation/d/1iAWApMInn4Ae8Y5P7D8Xkj9rjh_qJXObQHEVAMdqc9A/edit#slide=id.gf6049552c_0_0]
is updated with the latest design changes.
Some of the key changes include:
* Snapshot Edge is stored as key value (opposed to sorted set) since it will
not have multiple values.
* An Index Edge value consists of a Qualifier, non-indexed properties, and the
length of Qualifier.
- In HBase, A Qualifier of an Index Edge includes information such as index
prop. values, target vertex ids, and op. codes. Since Redis is not a
column-oriented storage, the same information is prepended to the value with an
additional timestamp for default timestamp ordering. For lack of a better term,
we call it Qualifier :).
- Length of the Qualifier is appended at the end of the value.
- Edge degree is handled by ZCARD.
* Vertex Edge updates are managed with an index field at the end of the value.
We also have a couple of issues that should be discussed:
* Table not supported: Originally, we considered keeping a virtual Table
information in the key since Redis Cluster does not support multiple tables.
However, performance issues when dropping all records in a table was a concern
so decided not worry about Tables too much during POC stage.
* ```incrementCounts``` is not yet implemented: This operation is used
exclusively by S2Counter. With the current Redis data model, incrementCounts
seemed like too much complication. But, I think we can overcome this by
utilizing the original S2Counter design which also used Redis.
> Add Redis storage(Key/Value storage) engine for S2Graph
> -------------------------------------------------------
>
> Key: S2GRAPH-1
> URL: https://issues.apache.org/jira/browse/S2GRAPH-1
> Project: S2Graph
> Issue Type: Task
> Reporter: Junki Kim
> Assignee: Junki Kim
> Labels: features
>
> h2. Background
> The purpose of S2Graph is to support all features of edge/vertex operations
> with HBase. It is also to orchestrate with HBase’s key features such as scale
> out / fault tolerant / highly available / etc. However, if there is no
> engineer to support stably in managing Hadoop cluster, it is very difficult
> to operate and maintain HBase and Hadoop clusters. Some companies, which
> already have another key/value storages to serve their services, may want to
> use S2Graph and also maintain their existing key/value storage environment.
> With S2Graph’s persistent storage, we would have a better support in
> operating another key/value storage. This is because S2Graph already has a
> strong feature, which can support multiple storage engines.
> What is best key/value storage engine starting to support S2Graph?
> Can companies maintain their existing key/value storage engine and use
> S2Graph at the same time? Yes, they can.
> Most of companies already have cache layers using Redis or Memcache. It is ok
> for the companies to start with one of these storages to support another
> key/value storage engine. It is because Redis is already suitable to support
> HA and fault tolerant features. Therefore, they do not need to replace this
> existing key/value storage engine.
> h2. Requirements
> - Support S2Graph storage engine’s interface
> -- Insert/Delete/Increment/etc.
> -- Query
> - Support serializer/deserializer to make edge/vertex data suit to key/value
> storage’s schema ( e.g. byte array encoding/decoding )
> - Support idempotent feature (this is a similar feature to S2Graph)
> - Design saving schema for edge(IndexEdge, SnapshotEdge) / vertex
> - Support query builder for S2Graph’s getEdges syntax
> - etc.
> I would like to invite everyone to openly discuss about this. I will also
> make a to-do list before I carry this issue in depth. Your feedback is
> appreciated :)
> P.S.
> I couldn't fill some information about this issue, e.g. affected version, fix
> version, component, etc. Please help me to fill out these mandatory fields.
> In addition, this issue is umbrella issue for real subsequent tasks, so I
> would like to make this epic issue. Can anyone help me out? I am new to
> create apache jira issue :)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
