DOYUNG YOON created S2GRAPH-213: ----------------------------------- Summary: Abstract Query/Mutation from Storage. Key: S2GRAPH-213 URL: https://issues.apache.org/jira/browse/S2GRAPH-213 Project: S2Graph Issue Type: Improvement Components: s2core Reporter: DOYUNG YOON
Currently, Storage has many components to implement, and most of them are very specific to HBase storage backend(ex: WriteWriteConflictResolver). Even though it is possible to add new storage backend using current abstraction, Storage, there are too many works which is not common to all storage backend I am suggesting following simple interface which is all common for any storage backend to be integrated with. {noformat} trait Fetcher { def init(config: Config)(implicit ec: ExecutionContext): Future[Fetcher] = Future.successful(this) def fetches(queryRequests: Seq[QueryRequest], prevStepEdges: Map[VertexId, Seq[EdgeWithScore]])(implicit ec: ExecutionContext): Future[Seq[StepResult]] def close(): Unit = {} } {noformat} {noformat} trait Mutator { def mutateVertex(zkQuorum: String, vertex: S2VertexLike, withWait: Boolean)(implicit ec: ExecutionContext): Future[MutateResponse] def mutateStrongEdges(zkQuorum: String, _edges: Seq[S2EdgeLike], withWait: Boolean)(implicit ec: ExecutionContext): Future[Seq[Boolean]] def mutateWeakEdges(zkQuorum: String, _edges: Seq[S2EdgeLike], withWait: Boolean)(implicit ec: ExecutionContext): Future[Seq[(Int, Boolean)]] def incrementCounts(zkQuorum: String, edges: Seq[S2EdgeLike], withWait: Boolean)(implicit ec: ExecutionContext): Future[Seq[MutateResponse]] def updateDegree(zkQuorum: String, edge: S2EdgeLike, degreeVal: Long = 0)(implicit ec: ExecutionContext): Future[MutateResponse] def deleteAllFetchedEdgesAsyncOld(stepInnerResult: StepResult, requestTs: Long, retryNum: Int)(implicit ec: ExecutionContext): Future[Boolean] } {noformat} By abstracting query/mutation as above interface, it is possible to implement JDBCFetcher and JDBCMutator that read/write vertex and edge into any JDBC enabled storage only implementing above interfaces. One thing to discuss is how we are going to maintain the information about what ServiceColumn/Label use which storage implementation. The naive solution would be store configuration for storage backend into ServiceColumn/Label's options field, which accepts JSON, and make S2Graph instance to maintain the mapping of what ServiceColumn/Label use which storage implementation. I think above abstraction make it possible to use different implementation per each ServiceColumn/Label, and more importantly, the user can provide their own implementation. For example, storing `User` vertex into Postgresql and `Friends` edges into HBase can be possible. Also, users who do not want to use S2Graph for vertex, do not need to store vertex at all, but by implementing `Fetcher` interface they can still traverse vertex as they are stored in S2Graph. Come up with this suggestion while working on S2GRAPH-206, since model serving requires different implementation for `Fetcher` per model. -- This message was sent by Atlassian JIRA (v7.6.3#76005)