[jira] [Commented] (S4-137) Add HDFS StateStorage for more reliable checkpoint saving
[ https://issues.apache.org/jira/browse/S4-137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13761229#comment-13761229 ] Kurtt.Lin commented on S4-137: -- @Matthieu, thanks for your comments. Indeed HDFSStateStorage is not very suited to large-sized, frequent checkpointing, as you point out. I've almost get the code done, but cannot decide the Hadoop core and confs should be included in the JUnit tests or not(I mean, they are kind of "heavy"). In addition, should pick Hadoop client 1.x or 2.x lib to go with the dev code? > Add HDFS StateStorage for more reliable checkpoint saving > - > > Key: S4-137 > URL: https://issues.apache.org/jira/browse/S4-137 > Project: Apache S4 > Issue Type: New Feature >Affects Versions: 0.6 >Reporter: Kurtt.Lin >Priority: Minor > Labels: checkpoint > Fix For: 0.7 > > Original Estimate: 120h > Remaining Estimate: 120h > > Currently DefaultFileSystemStateStorage is the only implementation for > StateStorage. Should have a storage like HDFS to improve the reliability of > the checkpoints' states. Some apps would benefit from this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (S4-137) Add HDFS StateStorage for more reliable checkpoint saving
[ https://issues.apache.org/jira/browse/S4-137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13757604#comment-13757604 ] Mariano Valles commented on S4-137: --- I worked on something related to this back on S4 0.4. At that moment one of the alternatives was to use Redis as a checkpoint storage mechanism. https://github.com/s4/s4/blob/master/s4-core/src/main/java/org/apache/s4/ft/RedisStateStorage.java I worked on creating an append-only file similar to the DefaultFileSystemStateStorage but using one file instead of many. But this was more a proof of concept implementation. While doing this we also thought on using a Distributed File System such as NFS or similar and just using the DefaultFileSystemStateStorage on this path. This would improve the availability of the checkpoints. > Add HDFS StateStorage for more reliable checkpoint saving > - > > Key: S4-137 > URL: https://issues.apache.org/jira/browse/S4-137 > Project: Apache S4 > Issue Type: New Feature >Affects Versions: 0.6 >Reporter: Kurtt.Lin >Priority: Minor > Labels: checkpoint > Fix For: 0.7 > > Original Estimate: 120h > Remaining Estimate: 120h > > Currently DefaultFileSystemStateStorage is the only implementation for > StateStorage. Should have a storage like HDFS to improve the reliability of > the checkpoints' states. Some apps would benefit from this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (S4-137) Add HDFS StateStorage for more reliable checkpoint saving
[ https://issues.apache.org/jira/browse/S4-137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13756069#comment-13756069 ] Matthieu Morel commented on S4-137: --- Good idea especially if you have a use case for that. You might also want to consider the induced latency though, depending on the checkpoint size and frequency. Make sure to use the hsync api for persistency guarantees. > Add HDFS StateStorage for more reliable checkpoint saving > - > > Key: S4-137 > URL: https://issues.apache.org/jira/browse/S4-137 > Project: Apache S4 > Issue Type: New Feature >Affects Versions: 0.6 >Reporter: Kurtt.Lin >Priority: Minor > Labels: checkpoint > Fix For: 0.7 > > Original Estimate: 120h > Remaining Estimate: 120h > > Currently DefaultFileSystemStateStorage is the only implementation for > StateStorage. Should have a storage like HDFS to improve the reliability of > the checkpoints' states. Some apps would benefit from this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (S4-137) Add HDFS StateStorage for more reliable checkpoint saving
I worked on something related to this back on S4 0.4. At that moment one of the alternatives was to use Redis as a checkpoint storage mechanism. https://github.com/s4/s4/blob/master/s4-core/src/main/java/org/apache/s4/ft/RedisStateStorage.java I worked on creating an append-only file similar to the DefaultFileSystemStateStorage but ysing one file instead of many. But this was more a proof of concept implementation. While doing this we also thought on using a Distributed File System such as NFS or similar and just using the DefaultFileSystemStateStorage on this path. This would improve the availability of the checkpoints. On Mon, Aug 19, 2013 at 4:33 AM, Kurtt.Lin (JIRA) wrote: > > [ > https://issues.apache.org/jira/browse/S4-137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13743490#comment-13743490] > > Kurtt.Lin commented on S4-137: > -- > > guys, please comment. can assign this to me. :) > > > Add HDFS StateStorage for more reliable checkpoint saving > > - > > > > Key: S4-137 > > URL: https://issues.apache.org/jira/browse/S4-137 > > Project: Apache S4 > > Issue Type: New Feature > >Affects Versions: 0.6 > >Reporter: Kurtt.Lin > >Priority: Minor > > Labels: checkpoint > > Fix For: 0.7 > > > > Original Estimate: 120h > > Remaining Estimate: 120h > > > > Currently DefaultFileSystemStateStorage is the only implementation for > StateStorage. Should have a storage like HDFS to improve the reliability of > the checkpoints' states. Some apps would benefit from this. > > -- > This message is automatically generated by JIRA. > If you think it was sent incorrectly, please contact your JIRA > administrators > For more information on JIRA, see: http://www.atlassian.com/software/jira >
[jira] [Commented] (S4-137) Add HDFS StateStorage for more reliable checkpoint saving
[ https://issues.apache.org/jira/browse/S4-137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13743490#comment-13743490 ] Kurtt.Lin commented on S4-137: -- guys, please comment. can assign this to me. :) > Add HDFS StateStorage for more reliable checkpoint saving > - > > Key: S4-137 > URL: https://issues.apache.org/jira/browse/S4-137 > Project: Apache S4 > Issue Type: New Feature >Affects Versions: 0.6 >Reporter: Kurtt.Lin >Priority: Minor > Labels: checkpoint > Fix For: 0.7 > > Original Estimate: 120h > Remaining Estimate: 120h > > Currently DefaultFileSystemStateStorage is the only implementation for > StateStorage. Should have a storage like HDFS to improve the reliability of > the checkpoints' states. Some apps would benefit from this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
