nwangtw commented on a change in pull request #2891: Refactor StatefulStorage
API
URL: https://github.com/apache/incubator-heron/pull/2891#discussion_r187484567
##########
File path: heron/proto/ckptmgr.proto
##########
@@ -171,7 +175,26 @@ message CleanStatefulCheckpointResponse {
repeated string cleaned_checkpoint_ids = 2;
}
-// stmgr -> ckptmgr messages
+/*
+ * stmgr -> ckptmgr messages
+ */
+
+// This message encapsulates the info associated with
+// state of an instance/partition
+message InstanceStateCheckpoint {
+ required string checkpoint_id = 1;
+ required bytes state = 2;
+ // A version string that can be specified by user. It can be used to
translate
Review comment:
We are thinking about calculation change. Not sure how useful it is though.
Please feel free to let us know if you feel this is overkill.
Normally the calculation in bolts won't change much and the version number
doesn't need to change. However in case the equation is changed, for example,
the old value is a duration in seconds and it is changed to duration in
millisecond, or it is possible the value is a data structure and a new field is
added. User can increase the version number and provide a conversion function
between the two versions. Then the topology would be able to load the stateful
data stored before the update, or skip/reject the data if topology doesn't know
how to convert between the two versions.
If topology cant tell the difference and just load the data, the state could
be polluted without user knowing it. The current design seems to rely on users
manually removing the data? it could be risky and hard since the topology needs
to be running in order to do it I think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
