Github user arunmahadevan commented on the pull request:
https://github.com/apache/storm/pull/644#issuecomment-124635613
@ptgoetz to address the concern of recovery times for large files, can we
recommend (or auto enforce) the max size in "size based rotation" to a
reasonable threshold (say 1GB). As long as the files get rotated at a
reasonable threshold which can be copied over quickly there wont be issues.
Regarding time based rotation: It may be possible to support by guarding
all the operations with locks and then have the new file path updated in the
index along with rotation. (there might be other corner cases which needs to be
thought through).
However the issue I see with time based rotation is we don't have a control
of the file size and if users configure a daily rotation policy and the data
rate is high, we wont be able to recover such huge files in case of failure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
