Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/644#issuecomment-128140442
I did a quick pass through the code and it looks OK, but I have not looked
at it in great detail. I am not very happy with the limitations on which
rotation policy you can use, nor on the size limit. I would rather be correct
but slow by default in all cases, even if they set bad configs (> 1GB), and
give them the power to make it fast but wrong if they know the risks and can
deal with it. Also a hard coded 1GB limit seems a little strange. What if we
have a 10GigE connection or even infiniband for HDFS and all of the data
happens to be in memory. We could in theory have processed the 1GB in less
then a second, still painful but not the end of the world.
Why don't we want to support a time based rotation, that rotates at the end
of a batch after the time has passed instead of in the middle of the batch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
