nwangtw commented on a change in pull request #2891: [WIP] Refactor
StatefulStorage
URL: https://github.com/apache/incubator-heron/pull/2891#discussion_r186219947
##########
File path: heron/proto/ckptmgr.proto
##########
@@ -171,7 +175,25 @@ message CleanStatefulCheckpointResponse {
repeated string cleaned_checkpoint_ids = 2;
}
-// stmgr -> ckptmgr messages
+/*
+ * stmgr -> ckptmgr messages
+ */
+
+// This message encapsulates the info associated with
+// state of an instance/partition
+message InstanceStateCheckpoint {
+ required string checkpoint_id = 1;
+ required bytes state = 2;
+ // A version string that can be specified by user. It can be used to
translate
+ // checkpoint data from older versions to the latest version when necessary.
+ optional string data_version = 3;
+}
+
+// This message encapsulates the info associated with
+// checkpoint metadata of a component
+message CheckpointComponentMetadata {
+ required int32 parallelism = 1;
Review comment:
kk. component name could be useful for viewing data. Will add.
Firstly, stateful data is organized around components (the schema should be
the same for all instances/partitions of the component and could be different
between components). It is overall a 3-level tree structure: topology is the
root, component is the mid level and instance/partition is the leaf level. We
need to store component level data in storage.
Secondly, the parallelism value is important to detect the plan changes and
handle the change correctly. In our first state, we can drop the stateful data
when parallelism doesn't match. In future, we should handle repartition based
on the old and new parallelisms of the component.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
