Github user arunmahadevan commented on a diff in the pull request:
https://github.com/apache/storm/pull/644#discussion_r37049715
--- Diff:
external/storm-hdfs/src/main/java/org/apache/storm/hdfs/trident/HdfsState.java
---
@@ -136,44 +174,98 @@ public void run() {
private transient FSDataOutputStream out;
protected RecordFormat format;
private long offset = 0;
+ private int bufferSize = 131072; // default 128 K
- public HdfsFileOptions withFsUrl(String fsUrl){
+ public HdfsFileOptions withFsUrl(String fsUrl) {
this.fsUrl = fsUrl;
return this;
}
- public HdfsFileOptions withConfigKey(String configKey){
+ public HdfsFileOptions withConfigKey(String configKey) {
this.configKey = configKey;
return this;
}
- public HdfsFileOptions withFileNameFormat(FileNameFormat
fileNameFormat){
+ public HdfsFileOptions withFileNameFormat(FileNameFormat
fileNameFormat) {
this.fileNameFormat = fileNameFormat;
return this;
}
- public HdfsFileOptions withRecordFormat(RecordFormat format){
+ public HdfsFileOptions withRecordFormat(RecordFormat format) {
this.format = format;
return this;
}
- public HdfsFileOptions withRotationPolicy(FileRotationPolicy
rotationPolicy){
+ public HdfsFileOptions withRotationPolicy(FileRotationPolicy
rotationPolicy) {
this.rotationPolicy = rotationPolicy;
return this;
}
- public HdfsFileOptions addRotationAction(RotationAction action){
+ /**
+ * <p>Set the size of the buffer used for hdfs file copy in case
of recovery. The default
+ * value is 131072.</p>
+ *
+ * <p> Note: The lower limit for the parameter is 4096, below
which the
+ * option is ignored. </p>
+ *
+ * @param sizeInBytes the buffer size in bytes
+ * @return {@link HdfsFileOptions}
+ */
+ public HdfsFileOptions withBufferSize(int sizeInBytes) {
+ this.bufferSize = Math.max(4096, sizeInBytes); // at least 4K
+ return this;
+ }
+
+ @Deprecated
--- End diff --
As mentioned in the previous comments and also in the jira (STORM-837),
rotation actions are too generic and make the recovery difficult if the process
crash in the middle of the action. Hence its best to avoid rotation actions in
case exactly once semantics is expected. Will add a note in the README.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
